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1400  KEY  BOULEVARD  SUITE  B200 
ARLINGTON,  VA  22209-5144 


May  21,2009 


MEMORANDUM  FOR  NATIONAL  SECURITY  PERSONNEL  SYSTEM 

OVERARCHING  INTEGRATED  PRODUCT  TEAM 


Subject:  Internal  Program  Evaluation  Report  on  NSPS 

Attached  is  the  first  internal  program  evaluation  report  on  the  National  Security 
Personnel  System  (NSPS),  prepared  under  the  auspices  of  section  9901.107  of  the  NSPS 
regulation,  which  states  that  the  Department  will  evaluate  NSPS  regulations  and  their 
implementation.  The  Program  Executive  Office  (PEO),  on  behalf  of  the  Secretary, 
carries  out  the  evaluation  activities  as  part  of  the  PEO  mission. 

NSPS  was  designed  to  be  a  flexible  system  to  meet  the  wide  ranging  needs  of 
many  different  DoD  organizations  and  their  employees.  Spiral  implementation  has 
enabled  the  Department  to  learn  continuously  through  formal  and  informal  evaluation 
activities,  and  to  adjust  the  system  and  its  supporting  tools  along  the  way.  This 
evaluation  encompasses  NSPS  staffing,  pay,  and  performance  features,  as  they  were 
implemented  through  mid-2008  for  Spiral  1  organizations,  and  is  based  on  information 
captured  after  the  first  full  rating  cycle  and  first  full  year  of  operation  for  most  of  the 
Spiral  1  workforce.  The  report  is  intended  to  show  where  NSPS  stood  at  the  end  of 
Spiral  1  in  relation  to  its  Key  Performance  Parameters,  and  to  highlight  areas  for 
attention. 


SRA  International,  Inc.,  a  support  contractor  to  the  PEO,  carried  out  the 
evaluation.  The  firm  has  years  of  experience  in  evaluating  alternative  personnel  systems, 
including  DoD  personnel  demonstration  projects  for  both  the  laboratory  and  acquisition 
communities.  SRA  is  also  extremely  well-versed  in  NSPS  data  and  processes,  and  has 
supported  the  PEO  from  the  beginning  of  the  NSPS  program. 

The  evaluation  considered  implementation  program  activities;  analyzed 
performance  rating  and  pay  outcomes;  sampled  performance  plans  and  assessments; 
analyzed  workforce  opinions  in  2006-2008  Status  of  Forces  Surveys  of  DoD  civilians; 
and  assessed  the  practices  and  views  of  local  managers,  employees,  and  supervisors  at 
twelve  Spiral  1  organizations. 

The  report  illustrates  that  the  program  is  experiencing  issues  typical  of  major 
programs  involving  significant  systemic  change.  In  2008,  Spiral  1  organizations  were 
still  refining  their  implementation  and  the  system  was  still  stabilizing  after  Congressional 
changes.  Many  members  of  the  workforce  expressed  doubts  or  uncertainty  about  NSPS. 
The  report  highlights  issues  with  the  compensation  flexibilities  available  for  internal 
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placement  actions  (e.g.,  reassignments),  and  describes  problems  with  the  automated 
appraisal  tool,  the  quality  of  performance  plans  and  assessments,  pay  pool  process 
transparency,  and  raters'  confidence  about  their  role  in  performance  management.  The 
report  also  shows  that  NSPS  has  had  a  positive  effect  on  attracting  highly  skilled 
newcomers  to  the  Department,  and  highlights  concerted  program  activities  to  train  the 
workforce,  communicate  on  key  matters,  and  institute  fair  appraisal  and  payout  review 
processes. 


While  the  2008  findings  reflect  an  early  stage  of  implementation,  it  remains 
essential  that  organizations  operating  under  NSPS  continue  to  focus  on  leadership 
communication,  continued  training  and  retraining,  exposure  to  the  pay  pool  panel 
deliberation  process  for  raters,  and  careful  and  equitable  management  processes  in  all 
aspects  of  NSPS.  The  Department  has  already  acted  on  a  number  of  key  problem  areas  in 
the  report,  including:  release  of  an  improved  automated  appraisal  tool  for  the  Fiscal  Year 
2009  rating  cycle;  a  new  Implementing  Issuance  requirement  to  communicate  annual  pay 
pool  results  to  the  affected  workforce;  additional  guidance  on  the  prohibition  of  forced 
distribution;  and  guidance  on  managing  performance  for  deployed  civilians. 

The  primary  purpose  of  this  evaluation  report  is  to  inform  senior  leaders  and 
policymakers  within  the  Department  as  they  consider  changes  and  improvements  to 
NSPS  policies  and  practices.  This  report  will  be  shared  with  the  NSPS  Senior  Executive, 
members  of  the  NSPS  Overarching  Integrated  Product  Team,  and  Component  NSPS 
Program  Managers.  It  will  also  be  provided  to  the  Defense  Business  Board  task  group 
that  has  been  established  to  conduct  a  comprehensive  review  of  the  program. 


The  PEO  remains  committed  to  evaluating  and  assessing  NSPS  to  ensure  its 
design  and  implementation  reflect  a  personnel  system  that  is  mission-focused,  fair, 
transparent,  and  effective. 
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ES.l  Purpose  of  the  Report 

The  purpose  of  this  evaluation  report  is  to  ascertain  whether  NSPS  as  implemented  in  Spiral  1  is 
achieving  or  is  on  track  to  achieve  the  expectations  set  forth  by  key  performance  parameters 
(KPPs)  in  the  NSPS  Requirements  Document.  The  secondary  purpose  is  to  identify  aspects  of 
NSPS  and  its  operation  that  warrant  attention  and  possible  adjustments.  This  evaluation 
focuses  on  the  NSPS  personnel  system  as  it  has  been  operating,  with  emphasis  on  performance 
management  and  compensation,  which  touch  all  employees,  and  to  a  lesser  degree  on 
classification  and  staffing. 

DoD's  white  collar  workforce  entered  NSPS  in  segments,  or  spirals.  Spiral  1  was  subdivided  into 
three  parts.  The  first  conversion,  Spiral  1.1,  included  more  than  10,000  employees  and 
supervisors  from  organizations  in  the  Army,  Navy,  Air  Force,  and  Fourth  Estate.  Their  first  rating 
cycle  was  a  six-month  period  that  began  on  30  April  2006.  Spirals  1.2  and  1.3  began  in  October 
2006  and  March  2007  respectively,  with  NSPS  conversions  taking  place  over  several  months  for 
each.  Spiral  1.2  included  about  64,000  employees,  and  Spiral  1.3  included  almost  34,000 
employees.  This  second  NSPS  rating  period  ended  in  September  2007,  with  the  accompanying 
payout  cycle  completed  in  January  2008.  Spiral  2  converted  in  FY08. 

ES.2  Evaluation  Goals  and  Objectives 

This  report  aligns  with  the  NSPS  Evaluation  Plan  issued  by  the  Program  Executive  Office  (PEO) 
for  NSPS  on  30  June  2007.  The  NSPS  Evaluation  Plan  lists  five  KPPs,  which  are  capabilities  or 
characteristics  that  are  "so  significant  that  failure  to  meet  a  minimum  'threshold'  can  be  cause 
for  that  element,  concept  or  system  to  be  reevaluated,  or  the  program  to  be  reassessed  or 
terminated"  (Requirements  Document,  25  September  2004,  p.  5).  The  five  KPPs  are: 

■  High-Performing  Workforce  and  Management  -  whether  employees  and  supervisors  are 
compensated  and  retained  based  on  their  performance  and  contribution  to  mission. 

■  Agile  and  Responsive  Workforce  and  Management  -  whether  the  workforce  can  be  easily 
sized,  shaped,  and  deployed  to  meet  changing  mission  requirements. 

■  Credible  and  Trusted  System  -  whether  the  system  assures  openness,  clarity, 
accountability  and  adherence  to  the  public  employment  principles  of  merit  and  fitness. 

■  Fiscally  Sound  System  -  whether  aggregate  increases  in  civilian  payroll,  at  the 
appropriations  level,  will  conform  to  OMB  fiscal  guidance,  and  whether  managers  will 
have  flexibility  to  manage  to  budget  at  the  unit  level. 

■  Supporting  Infrastructure  Information  Technology  Support  -  whether  information 
technology  support,  and  training  and  change  management  plans  are  available  and 
funded. 

ES.3  Evaluation  Data  Sources  and  Findings 

This  evaluation  report  analyzes  data  from  three  primary  sources: 

■  Automated  data  extracts  from  the  Defense  Civilian  Personnel  Data  System  (DCPDS)  and 
the  Compensation  Workbench  (CWB).  Section  2  presents  analyses  that  examine  rating 
and  payout  information  for  the  2007  performance  management  cycle  that  involved 
108,758  individuals.  Additional  analyses  in  this  section  address  2007  and  2008  personnel 
movement  among  NSPS  organizations,  out  of  NSPS,  or  out  of  DoD;  discrimination 
complaints;  and  sampled  employee  performance  plans. 
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■  Site  visits  to  12  organizations  across  all  four  components  and  operational  echelons.  The 
study  teams  conducted  58  focus  groups  and  53  interviews  and  reviewed  documentation 
relating  to  each  organization's  implementation  of  NSPS.  Section  3  addresses  the  site  visit 
findings. 

■  Employee  attitudes  as  reported  in  the  Status  of  Forces  Survey-Civilian  (SOFS-C)  for  2006, 
2007,  and  2008.  The  SOFS-C  is  administered  yearly  by  the  Defense  Manpower  Data 
Center.  Analyzing  these  responses  over  time  tracks  changes  in  workforce  attitudes 
regarding  satisfaction  with  job,  work,  workplace,  and  coworkers;  satisfaction  with 
leadership,  supervisors,  and  policies;  satisfaction  with  performance  management; 
satisfaction  with  career  progression;  and  attitudes  towards  NSPS.  Section  4  examines 
SOFS-C  responses. 

ES.4  Findings  bv  KPP 

Section  5  of  the  report  presents  an  integrated  summary  across  all  three  data  sources.  Key 
findings  presented  in  the  integrated  summary  indicate  that  while  NSPS  has  not  yet  achieved  all 
five  KPPs,  progress  is  being  made.  This  report  contains  key  findings  that  apply  to  the  five  KPPs 
as  well  as  three  findings  that  apply  to  NSPS  more  generally.  The  latter  are  listed  as  "Additional 
Topics." 

KPP  1.  High  Performing  Workforce  and  Management 

■  Stronger  link  between  total  compensation  and  performance.  The  link  between  pay  and 
performance  is  still  vague  for  many  employees  and  is,  in  the  opinion  of  some  senior 
leaders,  not  yet  fully  realized.  Analyses  of  DCPDS  data  show  a  clear  link  between 
performance  ratings  and  payouts.  According  to  SOFS-C  results,  a  recent  positive  upswing 
in  workforce  opinion  that  NSPS  has  improved  linking  pay  to  performance  is  offset  by  a 
greater  majority  indicating  a  contrary  perspective.  Employees  indicate  more  unfavorable 
than  favorable  views  that  NSPS  is  better  for  recognition  and  rewards  and  has  rewarded 
good  work  performance.  However,  all  NSPS  spirals  are  substantially  more  favorable  and 
less  unfavorable  than  the  non-NSPS  population  that  pay  raises  depend  on  how  well 
employees  perform. 

■  Market  sensitive  salary  and  wages.  NSPS  employees  are  generally  satisfied  with  pay,  but 
caution  that  the  five  percent  reassignment  cap  and  control  points  cause  concerns.  They 
also  express  concern  that  outside  hires  benefit  more  than  they  do  from  flexible  starting 
salaries.  Some  also  perceive  that  NSPS  funding  is  not  equivalent  to  GS  funding,  thereby 
potentially  limiting  their  earnings  and  affecting  retention. 

■  Performance  system  provides  ongoing  feedback.  Findings  from  employee  focus  groups 
generally  agree  with  SOFS-C  results  that  regular  and  formal  feedback  and  the  time 
required  to  develop  performance  plans  are  worthwhile,  although  SOFS-C  results  show  no 
strong  indication  that  NSPS  has  improved  communications  between  supervisors  and 
employees.  While  SOFS-C  results  show  employees  are  satisfied  with  their  performance 
recognition,  they  also  indicate  that  meaningful  distinctions  in  performance  are  not  readily 
seen  under  NSPS.  This  view  aligns  somewhat  with  focus  group  findings  that  some 
employees  do  not  agree  that  ratings  match  performance. 

■  Stronger  link  between  retention/separation  and  performance.  In  2008  employees  with 
low  ratings  separated  from  both  NSPS  and  non-NSPS  organizations  at  a  similar  rate  of 
about  13%.  Many  NSPS  employees  worry  that  NSPS'  lower  performance  ratings  will  make 
it  difficult  for  them  to  transfer  or  will  otherwise  negatively  affect  their  career  progression. 

■  Performance  expectations  aligned  with  DoD  and  organization  strategic  plans.  Findings 
from  SOFS-C  and  DCPDS  consistently  show  that  managers  communicate  goals  and 
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priorities.  SOFS-C  results  show  increasing  disagreement  concerning  alignment  to 
organizational  goals,  with  focus  group  findings  showing  more  general  agreement  that 
alignment  is  occurring. 

■  System  supports  employee  engagement  in  a  positive  manner.  SOFS-C  results  show  that 
employees  are  substantially  more  favorable  that  unfavorable  to  senior  leaders'  policies 
and  practices,  and  that  they  have  trust  and  confidence  in  both  their  supervisors  and  in 
senior  leaders.  Employee  focus  group  findings  indicate  that  NSPS'  wide  pay  bands  are 
appreciated  for  the  opportunities  they  provide  to  the  employee. 

KPP  2.  Agile  and  Responsive  Workforce  and  Management 

m  Renewable,  expandable,  retractable  (size)  -  mission  critical/core  support  occupations 
SOFS-C  findings  generally  show  some  increase  in  satisfaction  with  the  performance  and 
quality  of  employees  that  have  been  hired  in  the  past  two  years,  although  focus  groups  of 
supervisors  counter  this  perspective.  All  results  point  to  a  reluctance  to  credit  NSPS  for 
any  improvement  in  hiring  or  the  performance  of  new  hires  to  this  point.  Focus  group 
findings  cite  hiring  for  specific  pay  ranges  and  specialty  positions  as  particularly  difficult 
under  NSPS. 

■  Assignable,  deployable  (shape).  NSPS  has  flexibilities  that  are  recognized  by  both 
employees  and  supervisors,  but  SOFS-C  results  indicate  that  they  generally  see  NSPS  as 
worse  than  the  GS  system  for  hiring,  placement,  and  promotion.  A  key  issue  is  that  many 
employees  with  potential  to  be  supervisors  are  reluctant  to  be  reassigned  to  such 
positions  with  a  greater  workload  under  NSPS  and  at  most  a  five  percent  pay  increase. 

KPP  3.  Credible  and  Trusted  System 

m  All  elements  of  NSPS  design  are  accessible  and  understandable  (transparency).  Although 
all  organizations  established  pay  pool  business  rules  and  a  majority  published  payout  and 
rating  results,  many  employees  and  some  supervisors  do  not  fully  understand  the  pay 
pool  panel  process  and  wish  to  learn  more  about  it.  More  employees  disagree  than  agree 
that  the  pay  pool  panels  help  ensure  fair  ratings  and  payouts.  SOFS-C  and  focus  group 
findings  disagree  concerning  whether  employees  understand  what  is  required  to  be  rated 
at  a  different  performance  level. 

■  Management  decisions  are  merit-based,  non-discriminatory,  and  consistent  with  NSPS 
regulations.  All  NSPS  spirals  were  more  favorable  than  unfavorable  in  their  views  that 
promotions  in  their  work  units  are  based  on  merit.  As  well,  making  fair  personnel 
decisions  was  cited  as  the  most  important  supervisor  skill  under  NSPS.  All  three  Spiral  1 
populations  had  slightly  higher  unfavorable  than  favorable  response  rates  that  NSPS  is 
better/worse  than  the  previous  performance  management  system. 

■  Expeditious  and  fair  resolution  of  grievances,  performance  reconsiderations,  classification 
appeals,  adverse  action  appeals.  While  some  employees  did  not  request  reconsiderations 
because  they  did  not  know  how,  for  others  the  time  period  for  requesting  a 
reconsideration  was  an  issue,  or  they  were  waiting  another  pay  cycle  to  gain  perspective 
on  the  distribution  of  ratings.  This  lack  of  understanding  appears  to  be  substantiated  by 
pay  pool  managers  and  Performance  Review  Authorities  (PRAs)  who  report  a  lack  of 
guidance  on  how  to  conduct  the  reconsideration  process.  In  January  2008,  2.3%  of  the 
100,465  NSPS  employees  who  received  a  performance  rating  for  FY2007  requested  a 
reconsideration;  one-third  of  reconsideration  requests  were  granted.  Senior  leaders 
interpret  lack  of  reconsideration  requests  as  evidence  that  employees  are  satisfied  with 
the  pay  pool  panel  process.  The  total  number  of  EEO  complaints  increased  from  22  in 
2007  to  154  in  2008,  but  declined  as  a  percentage  of  those  in  NSPS.  However,  sufficient 
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data  are  not  yet  available  to  provide  a  meaningful  analysis  of  the  impact  of  NSPS  on 
discrimination  complaints. 

■  Handle  performance  deficiencies  and  misconduct  timely  and  decisively.  Supervisors  and 
managers  have  difficulty  in  dealing  with  poor  performers.  Many  employees  observed  that 
marginal  performers  are  seldom  given  2  ratings,  and  senior  leaders  and  supervisors 
generally  agree  that  there  is  institutional  difficulty  in  giving  2  ratings.  While  responses  on 
the  SOFS-C  were  generally  positive  about  steps  taken  in  their  work  unit  to  deal  with  a 
poor  performer,  employees  were  unwilling  to  credit  NSPS  with  improving  such  situations. 
Unfavorable  SOFS-C  responses  concerning  NSPS'  improving  disciplining/correcting  poor 
work  performance  exceed  favorable  ones.  All  spirals  expressed  general  ambivalence 
concerning  whether  NSPS  is  better  or  worse  than  their  previous  system  for  dealing  with 
conduct  and  discipline. 

■  NSPS  employees  in  aggregate  are  not  disadvantaged  by  the  compensation  system.  Rating 
and  payout  analyses  suggest  that  payouts  are  higher  for  higher-paid  employees.  The 
focus  groups  indicated  that  employees  are  concerned  about  including  part  of  the 
government-wide  pay  increase  (GPI)  in  pay  pool  performance  budgets,  the  effects  of 
bonuses  on  retirement  "high  three"  calculations,  and  how  they  fared  relative  to  GS 
employees. 

KPP  4.  Fiscally  Sound  System 

■  System  provides  for  cost  discipline.  Some  employees  in  organizations  that  enriched 
funding  to  their  NSPS  pay  pools  question  whether  the  higher  funding  levels  can  be 
sustained  over  the  long  term.  In  aggregate,  pay  pools  underspent  the  funding  for  salary 
increases  (Element  1  and  Element  2)  by  3.1  percent  and  overspent  the  funding  for 
bonuses  (Element  3)  by  5.5  percent.  The  underspending  of  Element  1  largely  reflects  non- 
rated  individuals  who  were  not  eligible  for  payouts.  The  total  payouts  were  lower  by  0.2 
percent  of  the  funding  amount. 

KPP  5.  Supporting  Infrastructure 

•  Employees  and  managers  are  knowledgeable  in  pertinent  aspects  of  NSPS  as  a  result  of 
training.  The  NSPS  workforce  seeks  additional  training  on  the  performance  appraisal 
system,  including  writing  performance  objectives  and  assessments.  Employees  and 
supervisors  also  expressed  a  desire  for  better  understanding  of  the  pay  pool  panel 
process.  Supervisors  are  increasingly  confident  that  they  have  the  tools,  training,  and 
information  they  need  to  make  pay  decisions  under  NSPS. 

Additional  Topics 

m  Amount  of  time  required  to  perform  process.  Supervisors  and  employees  overwhelmingly 
expressed  that  all  aspects  of  NSPS  were  time-intensive  in  the  first  year.  Focus  group 
findings  show  concern  and  uncertainty  whether  the  time  and  administration 
requirements  of  NSPS  will  be  worthwhile. 

■  Operation  and  administration  of  implementation  team.  Focus  group  findings  show  that, 
while  implementation  teams  were  instrumental  to  success,  their  efforts  were  confounded 
by  shifting  policies  during  the  initial  implementation  of  NSPS. 

■  Effects  of  NSPS.  While  focus  groups  are  divided  on  whether  the  workplace  is  more 
collaborative  or  competitive  under  NSPS,  SOFS-C  results  show  that  more  employees  feel 
their  co-workers  cooperate  to  get  the  job  done.  Employee  and  supervisor  focus  groups  as 
well  as  senior  leader  interview  findings  suggest  that  all  groups  believe  NSPS 
implementation  and  acceptance  will  take  time,  effort,  training,  and  leadership. 
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These  findings  are  often  inconsistent  across  data  sources.  This  result  is  not  entirely  surprising 
as  the  data  sources  measure  different  concepts  in  different  ways.  The  DCPDS  data  reflect 
empirical  results  for  all  of  NSPS.  The  SOFS-C  results  are  statistical  inferences  of  perceptions 
from  a  sample  of  NSPS  employees  (as  well  as  non-NSPS  civilians).  The  focus  groups  are 
protocol-driven  discussions  with  NSPS  employees  and  supervisors  about  their  experiences  and 
perceptions,  and  the  interviews  are  with  senior  managers  and  executives.  Given  the  stage  of 
NSPS  during  which  these  data  collection  efforts  occurred,  it  is  certainly  too  early  to  say  that 
NSPS  has  achieved  the  KPPs,  but  it  is  also  unreasonable  to  say  definitively  whether  NSPS  is  on 
or  off  track.  There  are  signs  of  progress  and  other  signs  of  needed  change. 

ES.5  Recommendations 

Many  of  the  study's  recommendations  address  communication.  Without  effective 
communication  and  training,  NSPS  cannot  achieve  its  goal  of  being  a  credible  and  trusted 
system.  Implementation  teams  were  conscientious  in  their  communication  efforts  -  so  much 
so  that  some  employees  reported  being  overloaded  with  frequent,  lengthy  e-mails  that  they 
didn't  have  time  to  read.  Further,  NSPS  as  a  new  system  continues  to  change,  and  some 
policies,  such  as  those  governing  use  of  contributing  factors,  vary  in  their  implementation. 
Policy  changes,  tool  updates,  new  employees  and  supervisors,  rotating  military  supervisors,  and 
lingering  questions  among  those  who  have  experienced  NSPS  all  call  for  high  levels  of  ongoing 
training  and  communication.  Key  recommendations  include: 

■  More  training  on  the  performance  management  system,  especially  writing  SMART 
objectives,  working  with  supervisors  to  tailor  objectives  that  apply  broadly  to  groups  or 
tasks,  changing  objectives  to  keep  them  aligned  with  changing  assignments,  writing  self- 
assessments,  and  understanding  use  of  performance  indicators  and  benchmarks. 
Supervisors  also  require  training  on  developing  objectives  and  writing  assessments  that 
enable  them  to  meet  the  needs  of  the  pay  pool  panel  and  defend  their  ratings.  Training 
must  include  hands-on  objective  writing  that  is  specific  to  the  participant's  job. 
Inventories  of  SMART  objective  examples  that  cover  a  range  of  job  types  and  levels  have 
proved  helpful.  Best  practices,  such  as  keeping  logs  or  journals  to  document 
accomplishments  and  results,  should  be  shared. 

■  High-level  training  for  employees  and  supervisors  that  explains  the  pay  pool  process, 
including  business  rules  and  other  checks  that  ensure  process  compliance  and 
accountability.  This  training  should  extend  through  the  PRA's  review  and  approval  of 
results.  It  should  also  include  a  high-level  description  of  the  Pay  Pool  Analysis  Tool  (PAT) 
that  allows  the  PRA  to  examine  the  organization's  rating  and  payout  results  across  pay 
pools  and  by  career  group,  pay  schedule,  and  pay  band.  The  workforce  should  also  know 
that  the  components,  as  well  as  the  NSPS  PEO,  conduct  analyses  of  ratings  and  payouts 
for  protected  groups. 

■  Easy  reference  guides,  check  lists,  process  flow  graphics,  time  lines,  and  other  learning 
aids  that  employees  can  keep  at  hand  and  refer  to  quickly. 

■  Mock  pay  pools,  which  participants  have  found  valuable,  as  refreshers  for  continuing 
panel  members  and  as  training  for  new  members.  This  training  and  other  panel  planning 
meetings  led  by  the  pay  pool  manager  are  important  for  fostering  a  common 
understanding  and  application  of  the  performance  indicators  and  benchmarks,  including 
how  expectations  should  rise  with  the  level  of  work.  This  understanding  of  ratings  and 
performance  indicators  must  be  shared  with  the  managers  and  supervisors  whose 
employees  constitute  the  pay  pool,  as  well  as  with  employees  themselves. 

■  Improved  feedback  for  effective  performance  management.  Managers  and  supervisors 
need  coaching  in  how  to  deliver  useful  feedback,  especially  during  the  interim  and  year- 
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end  reviews,  but  also  in  response  to  successes,  failures,  or  missed  opportunities 
whenever  they  occur.  Further,  employees  should  be  encouraged  to  seek  feedback  if  they 
need  but  don't  receive  it. 

■  Greater  clarity  on  career  progression  and  compensation  policy  to  eliminate  confusion  for 
the  workforce  that  has  moved  from  the  comfortable,  well-understood  GS  structure  to 
NSPS  pay  bands.  Policies  or  business  rules  governing  reassignments  that  merit  up  to  a 
five  percent  increase,  where  control  points  apply,  how  performance  expectations  change 
as  the  level  of  work  increases,  rounding  rules,  and  salary-bonus  splits  all  merit  wide 
dissemination  and  discussion.  These  are  issues  on  which  questions  of  fairness  invariably 
arise  if  employees  sense  that  there  is  no  policy  or  that  the  policy  is  complied  with 
selectively.  Among  these,  the  five  percent  limit  on  pay  increases  for  reassignments  is 
perhaps  most  opposed  or  misunderstood. 

ES.6  Conclusion 

The  KPPs  set  high  standards  for  NSPS,  but  certainly  standards  that  an  effective,  performance- 
based  personnel  system  should  achieve.  NSPS  implementation  made  great  progress  from  its 
beginning  in  April  2006  with  just  over  10,000  employees  to  the  conclusion  of  Spiral  1  with  a 
total  of  more  than  100,000.  Program  managers,  implementation  teams,  and  trainers  across 
DoD  worked  with  executives,  managers,  supervisors,  and  employees  to  make  NSPS  real.  DoD 
knows  from  its  decades  of  experience  with  performance-based  demonstration  projects  that 
implementing  and  adjusting  to  change  take  time,  especially  when  the  change  is  of  such 
magnitude  as  moving  many  thousand  employees  from  the  old  but  familiar  General  Schedule  to 
a  new  pay-for-performance  personnel  system.  Evaluations  of  DoD  demonstration  projects  have 
shown  that  negative  initial  responses  become  positive  over  time  as  new  employees  who  accept 
pay  for  performance  join  the  workforce,  those  who  most  resist  it  leave,  processes  become 
more  familiar  and  efficient,  and  high  performers  see  real  rewards.  This  Spiral  1  evaluation 
report  has  examined  NSPS  after  just  two  performance  cycles  for  Spiral  1.1  and  only  one  cycle 
for  Spirals  1.2  and  1.3.  The  report  describes  a  beginning  and  a  process,  not  an  arrival. 

Even  with  massive  training  and  communication  efforts,  the  workforce  has  unanswered 
questions.  Employees  and  supervisors  both  want  to  understand  the  performance  management 
system  better,  especially  what  happens  in  pay  pools  at  the  cycle's  critical  end.  The  vast 
majority  are  invested  in  their  work  and  want  feedback  to  help  improve  their  performance.  The 
workforce  found  the  process  of  writing  SMART  objectives  and  assessments  new  and 
challenging,  and  pay  pool  panels  wrestled  with  performance  plans  that  were  not  always  as 
specific  as  required.  Together,  they  made  it  work,  though  generally  not  as  smoothly  as  anyone 
would  have  wished.  There  were  changes  along  the  way.  Some  changes  were  the  result  of 
learning,  some  the  result  of  Congressional  decision  making  -  all  were  in  the  spirit  of 
improvement. 

The  way  is  paved  for  the  future.  Leadership  is  on  board,  the  training  is  established,  the  trainers 
are  trained,  and  the  Readiness  Tool  documents  lessons  learned.  Sustainment  training  will 
update  and  refresh  the  original  training,  improve  proficiency,  and  answer  such  lingering 
questions  as  whether  to  weight  objectives,  how  pay  pool  funding  compares  to  GS,  and  how 
business  rules  inform  share  allocation.  The  processes  that  make  up  the  annual  cycle  will 
become  more  familiar.  The  advantages  of  working  with  pay  bands  -  classification, 
reassignments,  career  progression  -  will  become  more  apparent  as  attachment  to  the  GS 
structure  fades.  As  the  newness  continues  to  wear  off,  NSPS  has  the  potential  to  demand  less 
time  and  effort  and  become  a  valuable  tool  for  both  supervisors  and  employees.  It's  on  its  way, 
and  future  evaluations  will  document  its  path. 
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1.1.  Background  and  Purpose 

To  support  America's  uniformed  combat  forces,  the  Department  of  Defense  (DoD)  employs 
more  than  700,000  appropriated  fund  civilians,  including  both  blue  collar  and  white  collar 
workers.  Now  faced  with  a  global  war  on  terrorism,  the  role  of  DoD's  civilian  workforce  is 
expanding  to  include  more  significant  participation  in  combat  support  functions  that  will  allow 
military  personnel  to  focus  on  war  fighting  duties.  In  addition,  civilian  personnel  are  key  to 
maintaining  institutional  knowledge  because  of  frequent  rotations  of  military  personnel. 

Since  the  end  of  the  Cold  War,  the  civilian  workforce  has  undergone  substantial  changes 
resulting  from  downsizing,  base  realignments  and  closures,  and  competitive  sourcing  initiatives. 
In  addition,  the  laws,  rules,  regulations,  policies,  and  procedures  for  hiring,  firing,  utilizing,  and 
paying  these  employees  have  evolved  over  several  decades,  resulting  in  complexity  and  rigidity 
in  civilian  personnel  processes. 

Based  on  DoD's  success  with  several  demonstration  projects  over  20  years,  Congress  passed 
Public  Law  108-136,  the  Defense  Authorization  Act  for  Fiscal  Year  2004.  Title  XI,  Subtitle  A, 
section  1101  of  this  law  gives  the  Secretary  of  Defense  authority  to  establish  a  flexible  and 
contemporary  system  of  civilian  Human  Resources  (HR)  management  for  DoD  civilians,  in 
coordination  with  the  Director,  Office  of  Personnel  Management  (OPM),  and  in  collaboration 
with  employee  representatives.  The  National  Security  Personnel  System  (NSPS)  enables  DoD  to 
"manage  strategically  its  civilian  workforce  based  on  a  total  force  perspective"  (Requirements 
Document  for  National  Security  Personnel  System,  25  September  2004,  p.  3)  and  to  better  meet 
the  national  security  demands  of  the  21st  century.  The  overarching  mission  objective  of  NSPS  is 
"to  place  the  right  civilian  employee  in  the  right  job  with  the  right  skills  at  the  right  time  at  the 
right  cost."  That  is,  "NSPS  must  be  a  mission-based  system  that  is  linked  to  DoD  goals.  The 
NSPS  system  must  allow  rapid  adaptation  of  the  civilian  workforce  composition  to  meet 
changes  in  mission  requirements"  (Requirements  Document  for  National  Security  Personnel 
System,  25  September  2004,  p.  3). 

DoD's  white  collar  workforce  is  entering  NSPS  in  segments,  or  spirals.  Spiral  1  was  subdivided 
into  three  spirals.  The  first  conversion,  Spiral  1.1,  included  more  than  10,000  employees  and 
supervisors  from  organizations  in  the  Army,  Navy,  Air  Force,  and  two  DoD  agencies.  Their  first 
rating  cycle  was  a  six-month  period  that  began  on  30  April  2006.  This  initial  rollout  did  not 
include  organizations  with  employees  covered  by  bargaining  units  or  employees  at  foreign 
locations.  Spirals  1.2  and  1.3  began  in  October  2006  and  March  2007,  respectively,  with  NSPS 
conversions  taking  place  over  several  months  for  each.  Spiral  1.2  included  about  64,000 
employees,  and  Spiral  1.3  totaled  almost  34,000  employees.  This  second  NSPS  rating  and 
payout  cycle  was  completed  in  January  2008.  Additional  spirals  converted  in  FY08  and  more 
are  planned  in  later  years. 

The  purpose  of  this  evaluation  report  is  to  ascertain  whether  NSPS  as  implemented  in  Spiral  1  is 
achieving  or  is  on  track  to  achieve  the  expectations  set  forth  in  its  key  performance  parameters 
in  the  NSPS  Requirements  Document.  The  secondary  purpose  is  to  identify  aspects  of  NSPS  and 
its  operation  that  warrant  attention  and  possible  adjustments.  This  evaluation  concentrates  on 
the  NSPS  human  resource  management  system  as  it  has  been  operating,  with  emphasis  on 
performance  management,  which  touches  all  employees,  and  to  a  lesser  degree  on 
compensation,  classification,  and  staffing.  This  report  is  a  response  to  the  NSPS  Evaluation  Plan 
issued  by  the  Program  Executive  Office  (PEO)  for  NSPS  on  30  June  2007. 
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1.2.  NSPS  Goals  and  Objectives 

In  designing  NSPS,  senior  leaders  adopted  a  set  of  Guiding  Principles  as  a  compass  to  direct 
efforts  throughout  all  phases  of  NSPS  development.  These  principles,  set  forth  in  the 
Requirements  Document  (25  September  2004,  p.  5),  translate  and  communicate  the  broad 
requirements  and  priorities  outlined  in  the  legislation  into  concise  statements  that  underscore 
DoD's  purpose  and  intent  in  creating  NSPS. 

The  NSPS  Guiding  Principles  are: 

■  Put  mission  first  -  support  National  Security  goals  and  strategic  objectives; 

■  Respect  the  individual  -  protect  rights  guaranteed  by  law; 

■  Value  talent,  performance,  leadership  and  commitment  to  public  service; 

■  Be  flexible,  understandable,  credible,  responsive,  and  executable; 

■  Ensure  accountability  at  all  levels; 

■  Balance  HR  interoperability  with  unique  mission  requirements;  and 

■  Be  competitive  and  cost  effective. 

Senior  leadership  also  approved  a  series  of  Key  Performance  Parameters  (KPPs).  According  to 
the  Requirements  Document  (25  September  2004,  p.  5),  "a  KPP  is  a  capability  or  characteristic 
that  is  so  significant  that  failure  to  meet  a  minimum  'threshold'  can  be  cause  for  that  element, 
concept  or  system  to  be  reevaluated,  or  the  program  to  be  reassessed  or  terminated."  Each  of 
the  KPPs  has  associated  objectives,  which  further  define  that  parameter  and  allow  it  to  be 
measured.  Table  1.1  summarizes  the  Key  Performance  Parameters. 


Table  1.1  -  Key  Performance  Parameters 


1.  High  Performing  Workforce 
and  Management 

Employees  and  supervisors  are  compensated  and  retained  based  on  their 
performance  and  contribution  to  mission 

2.  Agile  and  Responsive 

Workforce  and  Management 

Workforce  can  be  easily  sized,  shaped,  and  deployed  to  meet  changing  mission 
requirements 

3.  Credible  and  Trusted  System 

System  assures  openness,  clarity,  accountability  and  adherence  to  the  public 
employment  principles  of  merit  and  fitness 

4.  Fiscally  Sound  System 

Aggregate  increases  in  civilian  payroll,  at  the  appropriations  level,  will  conform 
to  OMB  fiscal  guidance;  managers  will  have  flexibility  to  manage  to  budget  at 
the  unit  level 

5.  Supporting  Infrastructure 

Information  Technology  support,  and  training  and  change  management  plans 
are  available  and  funded 

KPP  1:  High-Performing  Workforce  and  Management 

The  foundation  of  NSPS  is  that  employees  and  supervisors  must  be  compensated  and  retained 
based  on  their  performance  and  contribution  to  the  mission.  This  reward  system  must  apply 
not  only  to  individuals,  but  also  to  organizations  and  teams  of  employees.  Work  objectives  must 
be  clearly  defined  and  should  align  with  DoD  and  organization  strategic  plans  and  mission 
requirements.  The  system,  while  preserving  merit  principles  and  veterans'  preference 
entitlements,  must  be  based  on  simplified  personnel  rules  that  support  flexibility  and 
adaptability  to  varying  command  missions  and  structures.  Most  important  of  all,  the  system 
must  be  constructed  so  that  employees  can  readily  understand  how  and  why  performance 
ratings  are  assigned.  Employees  must  also  receive  clear  feedback  on  past  performance  and 
guidance  on  how  they  can  improve  their  performance  and  competencies  and  manage  their 
careers. 

The  attributes  of  KPP  1  are: 

■  System  is  transparent— clear  and  understandable  to  employee  and  supervisor  alike 

■  Credible  system— trusted  by  employees  and  supervisors 
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■  Performance  and  contribution  are  linked  to  salary  and  rewards 

■  Salary  and  rewards  enable  DoD  to  compete  successfully  in  hiring  and  retaining  employees 

■  System  links  to  the  DoD  and  Component  strategic  plans 

■  System  allows  for  variations  without  incurring  excess  cost  to  support  performance 
management  processes 

■  System  provides  ongoing  feedback 

■  System  is  contemporary 

KPP  2:  Agile  and  Responsive  Workforce  and  Management 

NSPS  must  enhance  the  Department's  ability  to  more  easily  size,  shape,  and  deploy  any 
member  of  the  workforce— whether  employee,  supervisor,  or  manager— to  meet  changing 
mission  and  stationing  requirements,  including  those  related  to  Joint  Operations.  Management 
actions  must  adhere  to  merit  principles,  veterans'  preferences,  and  employees'  rights  to 
organize  and  bargain  collectively. 

The  attributes  of  KPP  2  are: 

■  Expandable 

»  Workforce  additions  can  be  easily  made  to  meet  emergent  mission  requirements 
o  Flexibilities  exist  to  hire  or  expand  to  meet  fluctuating  work  loads 
°  Workforce  skills  are  readily  identifiable  in  order  to  assign  employees  to  meet 
emergent  mission  requirements 

■  Retractable 

o  Workforce  can  be  easily  right-sized  to  meet  decreased  mission  requirements 
°  System  is  compatible  with  competitive  sourcing  regulations  and  provides  the  flexibility 
to  create  and  compete  DoD  Most  Efficient  Organizations  (MEO)  within  the  A-76 
process 

°  System  supports  management  decisions  on  modifications  of  employee  numbers  while 
sustaining  a  core  group  (e.g.,  Temporary,  Term  and  Permanent  Employee  mix 
decisions) 

■  Assignable 

°  Employees  can  be  easily  assigned/reassigned  work  in  support  of  ongoing/emergent 
mission  requirements 

°  Employees  can  be  moved  within  a  broad  pay  band  vice  having  to  be  reassigned  or 
detailed  to  a  specific  series  and  pay  grade 
•  System  is  sufficiently  adaptable  to  allow  for  the  need  for  changing  skill  sets 

■  Deployable 

°  Employees  and/or  a  work  unit  can  be  easily  geographically  moved,  either  temporarily 
or  permanently,  to  meet  changing  mission  requirements,  including  Joint  requirements 
and  across  DoD  Component  structures 

°  Flexibility  exists  to  provide  incentives  for  employees  to  move  or  be  deployable 

■  Renewable 

°  The  system  provides  for  growth  and  sustainment  of  competencies  (i.e.,  knowledge, 
skills,  abilities,  behaviors)  throughout  a  career 
°  New  skills  and  talent  can  be  brought  into  the  organization  quickly  and  impartially 

°  Personnel  with  obsolete  skills  can  be  retrained  to  achieve  the  proper  skill  mix  needed 

to  meet  mission 

°  The  system  supports  and  facilitates  succession  planning 
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■  Reconfigurable 

°  Organizational  structures  can  be  easily  changed  to  meet  ongoing  and  emergent 
mission  requirements 

■  Diverse 

°  Workforce  is  representative  and  multi-skilled  with  varied  backgrounds  and 
experiences 

■  Contemporary 

°  System  changes  and  adapts  to  meet  current  needs  and  changing  conditions 
o  System  does  not  impose  unnecessary  rules  and  regulations  that  restrict  otherwise 
legally  permissible  management  action 

KPP  3:  Credible  and  Trusted  System 

The  performance  management  system  within  NSPS  must  include  a  fair,  credible,  and 
transparent  performance  appraisal  system,  timetables  for  review  of  employee  performance, 
dialogue  between  employees  and  supervisors,  and  an  appeals  process  for  equitably  and 
expeditiously  resolving  workforce  concerns.  The  NSPS  processes  must  be  easily  accessible  and 
understandable,  with  clear  articulations  of  each  employee's  role  and  responsibilities. 
Performance  expectations  and  corresponding  salary  and  bonuses  must  also  be  equitable  and 
clearly  understood.  A  labor-management  relationship  must  be  fostered  that  effectively 
addresses  employee  concerns  without  compromising  DoD  mission  accomplishment. 

The  attributes  of  KPP  3  are: 

■  System  design  is  accessible,  understandable,  accountable  and  merit-based 

■  System  provides  for  fair  and  expeditious  resolution  of  issues  and  concerns 

■  System  includes  a  performance  management  system  that  meets  statutory  requirements 

KPP  4:  Fiscally  Sound  System 

NSPS  development,  implementation,  and  life  cycle  maintenance  costs  must  be  funded  within 
the  DoD  top  line.  Once  NSPS  is  implemented  there  must  be  cost  discipline  such  that  aggregate 
increases  in  civilian  payroll  at  each  appropriation  level  conform  to  OMB  fiscal  guidance.  NSPS 
must  be  cost-neutral  so  that  in  the  aggregate  employees  are  not  disadvantaged. 
Managers/supervisors  must  be  trained  on  the  impact  of  their  fiscal  decisions  on  DoD  mission 
performance.  Management  authorities  should  support  managing  to  budget  at  the  unit  level 
while  maintaining  the  flexibility  to  offer  market  sensitive  pay.  The  NSPS  system  must  ensure 
people  are  not  compensated  outside  their  level  of  work  responsibility,  either  too  high  or  too 
low. 

The  attributes  of  KPP  4  are: 

■  Aggregate  increases  in  civilian  payroll  at  the  appropriation  levels  conform  to  OMB  fiscal 
guidance  and  statutory  requirements 

■  Funded  implementation  costs  are  measured  with  respect  to  the  DoD  top  line 

■  System  provides  for  cost  discipline 

■  System  provides  flexibility  to  manage  civilian  personnel  to  budget  at  the  unit  level 

KPP  5:  Supporting  Infrastructure 

NSPS  must  be  supported  by  a  robust  infrastructure  that  facilitates  user  operational  and 
functional  requirements.  That  infrastructure  must  include  an  interoperable  information 
technology  (IT)  system  that  is  easy  to  use;  accessible  to  all  users  with  appropriate  permissions; 
and  capable  of  generating  the  reports,  analyses  and  deliverables  necessary  for  all  types  of 
finance,  manpower  management,  HR  and  other  functional  requirements  and  for  evaluation  of 
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the  NSPS.  A  second  element  of  the  NSPS  infrastructure  is  a  pay  and  performance  management 
information  system  that  provides  sufficiently  robust  data,  reports,  and  transaction  processing 
to  support  sound,  fair  compensation  decisions.  Because  NSPS  represents  significant  change  in 
the  management  of  employees,  operations  and  systems  embedded  throughout  DoD, 
implementation  of  NSPS  requires  training  in  the  use  of  these  systems,  as  well  as  training  in 
communications,  change  management  and  the  "soft  skills"  associated  with  performance 
management  and  pay  setting;  and  in  all  aspects  of  NSPS  policy,  performance  management,  and 
operations. 

The  attributes  of  KPP  5  are: 

■  Employees  and  managers  are  knowledgeable  about  NSPS  as  a  result  of  training 

■  Supporting  infrastructure  provides  interoperability  across  all  offices  and  functions 

■  Data  are  accessible  when  personnel  possess  appropriate  permissions 

1.3.  Major  Features  of  NSPS 

NSPS  is  designed  to  promote  a  culture  in  which  the  performance  and  contributions  of  the  DoD 
civilian  workforce  are  more  fully  recognized  and  rewarded.  The  system  offers  the  civilian 
workforce  broad  pay  bands  with  performance-based  pay.  As  the  Department  moves  away  from 
the  General  Schedule  (GS),  it  will  be  able  to  adjust  salaries  in  response  to  factors  including  labor 
market  conditions,  performance,  and  changes  in  duties.  NSPS  helps  attract  skilled,  talented,  and 
motivated  people  while  also  helping  retain  and  encourage  improvement  in  the  skills  and 
mission  focus  of  the  current  workforce.  It  serves  as  the  foundation  for  a  leaner,  more  flexible 
support  structure.  NSPS  includes  new  provisions  for  classification,  pay/compensation,  and 
performance  management  systems,  as  well  as  new  flexibilities  for  hiring  and  staffing.  These 
features  of  NSPS  are  elaborated  in  the  sections  below. 

1.3.1.  Classification 

■  A  new  pay  banding  system  replaces  the  GS  system  and  provides  flexibility  to  assign  or 
assume  work  responsively  to  changing  mission  requirements,  new  technologies,  new 
employee  competencies,  and/or  systemic  approaches  to  work 

■  Positions  are  grouped  in  broad  career  groups;  associated  pay  schedules  are  based  on  the 
nature  of  the  work,  career  patterns,  and  competencies;  associated  pay  bands  replace  GS 
grades  with  broad  salary  ranges  for  broad  levels  of  work  (e.g.,  entry,  full  performance, 
expert) 

■  Lengthy,  detailed  job  descriptions  are  no  longer  needed 

1.3.2.  Pay/Compensation 

■  The  new  pay  system  is  intended  to  attract,  develop,  retain,  and  reward  high-performing 
employees  through  appropriate  compensation 

■  The  system  provides  the  framework  for  DoD  to  move  towards  market-sensitive  pay: 

°  Market  factors  are  considered  when  setting  pay  for  new  hires  and  job  changes 
»  Local  market  and  occupational  conditions  are  considered  in  rates  authorized  for  pay 
schedules  and  local  market  supplements 

■  Movement  through  a  pay  band  is  based  primarily  on  performance/contribution,  though 
increased  responsibility  or  job  complexity  also  can  be  compensated  in  band 

■  Annual  performance  pay  increases: 

°  Are  based  on  performance/contribution,  rather  than  longevity 
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°  Allow  larger  percentage  increases  to  be  given  to  outstanding  performers 
°  Are  not  given  to  employees  performing  below  the  fully  acceptable  level 

■  Adjustments  to  rate  ranges  are  determined  by  the  Secretary;  increases  associated  with 
rate  range  adjustments  are  not  given  to  unacceptable  performers 

■  Pay  schedule  ranges  are  not  bound  by  General  Schedule  levels 

1.3.3.  Performance  Management 

■  The  NSPS  performance  management  system  is  designed  to  encourage  employee 
engagement  on  behalf  of  the  organization's  mission  and  goals  through  output  and 
outcome  oriented  individual  performance  objectives  and  expectations 

■  Ongoing  communication  and  feedback  about  performance  will  occur  between  the 
supervisor  and  employees  in  setting  expectations  and  in  reviewing  and  assessing 
performance 

■  Ratings  will  reflect  meaningful  distinctions  in  employee  performance,  and  there  will  be  no 
forced  rating  distribution  or  quotas 

■  Conduct  on  the  job  (behavior,  professional  demeanor)  is  recognized  as  an  aspect  of 
performance 

■  Intense  training  in  the  NSPS  performance  management  system  and  related  skills  will  be 
provided  to  employees,  supervisors,  managers,  and  human  resource  practitioners  to 
ensure  understanding  of  and  responsible  action  under  the  system 

■  A  pay  pool  process  and  a  rating  reconsideration  process  will  be  established  to  foster 
fairness  and  equity 

1.3.4.  Hiring/Staffing 

m  NSPS  will  provide  flexibilities  to  augment  hiring  authorities,  provide  alternative  forms  of 
competition,  and  streamline  examining  procedures  while  respecting  the  principles  of 
merit  and  fitness  and  preserving  veterans'  preferences 

■  A  direct-hire  authority  for  severe  shortage  or  critical  needs  will  be  vested  in  the  Secretary 

1.3.5.  Career  Groups,  Pay  Schedules,  and  Pay  Bands 

The  white  collar  workforce  is  broken  out  into  four  career  groups  based  on  occupational  code: 
Standard  Career  Group  (about  72  percent  of  the  white  collar  workforce);  Scientific  and 
Engineering  Career  Group  (about  23  percent);  Medical  Career  Group  (about  2  percent);  and 
Investigative  and  Protective  Services  Career  Group  (about  3  percent).  Each  career  group  is 
further  broken  out  into  three  or  four  pay  schedules  and  each  pay  schedule  to  one  to  four  pay 
bands.  Table  1.2  summarizes  the  job  classification  architecture.  The  entries  in  the  cells  in  the 
band  columns  represent  the  GS  pay  range  or  2008  dollar  range  for  the  pay  bands. 
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Table  1.2  -  CY 2008  Pay  Ranges  Relative  to  General  Schedule  Pay  Tables 


Career  Group 

Pay  Schedule 

Pay  Range  (Relative  to  General  Schedule)  ]| 

Description 

Code 

Band  1 

Band  2 

Band  3 

Band  4  1 

Professional/Analytical 

YA 

5-1  to  11-10 

9-1  to  13-10 

13-5  to  15-10 

+5% 

Technician/Support 

YB 

1-1  to  6-10 

7-1  to  10-10 

11-1  to  12-10 

Standard 

Supervisor/Manager 

YC 

7-1  to  11-10 

12-1  to  14-10 

+  5% 

14-1  to  15-10 

+  5% 

Student  Educational 
and  Employment 

YP 

1-1  to  11-10 

Professional 

YD 

5-1  to  11-10 

9-1  to  13-10 

13-5  to  15-10 

+5% 

Scientific  and 

Technician/Support 

YE 

1-1  to  6-10 

7-1  to  10-10 

11-1  to  12-10 

12-5  to  13- 

Engineering 

Supervisor/Manager 

YF 

7-1  to  11-10 

12-1  to  14-10 

13-5  to  15-10 

10 

+  5% 

+5% 

Physician/Dentist 

YG 

$89,269  to 

$115,143  to 

$175,000 

$225,000 

Professional 

YH 

5-1  to  11-10 

9-1  to  14-10 

13-5  to  15-10 

Medical 

+5% 

Technician/Support 

Yl 

1-1  to  6-10 

7-1  to  10-10 

11-1  to  12-10 

Supervisor/Manager 

YJ 

7-1  to  11-10 

12-1  to  14-10 

+  5% 

14-1  to  15-10 

+5% 

$101,870  to 
$200,000 

Investigative 

YK 

5-1  to  11-10 

9-1  to  13-10 

13-5  to  15-10 

+5% 

Investigative 

Fire  Protection 

YL 

1-1  to  6-10 

7-1  to  10-10 

11-1  to  12-10 

12-5  to  14- 

and  Protective 
Services 

Police/Security  Guard 

YM 

1-1  to  6-10 

7-1  to  10-10 

10 

Supervisor/Manager 

YN 

7-1  to  11-10 

12-1  to  14-10 

14-1  to  15-10 

+  5% 

+5% 

1.3.6.  Link  Between  Pay  and  Performance 

NSPS  uses  pay  pools  to  administer  the  rating  and  performance  payout  processes.  Typically,  pay 
pools  are  based  on  organizational  structure  and  include  from  35  to  several  hundred  employees. 
Individual  employees  work  with  their  supervisors  to  develop  performance  plans  that  include 
from  one  to  ten  job  objectives  that  are  related  to  the  organization's  mission  and  goals.  Job 
objectives  are  used  to  communicate  major  individual,  team,  or  organizational  responsibilities  or 
contributions  and  related  outcomes  expected  of  the  employee  and  appropriate  for  the 
employee's  pay  band.  Contributing  factors  are  the  attributes  and  behaviors  that  identify  how 
the  critical  work  established  in  job  objectives  is  expected  to  be  accomplished. 

For  each  job  objective,  generally  at  least  one  of  the  following  contributing  factors  is  identified: 

■  Communication  ■  Leadership 

■  Cooperation  and  Teamwork  ■  Resource  Management 

■  Critical  Thinking  ■  Technical  Proficiency 

■  Customer  Focus 

Job  objectives,  contributing  factors,  and  objective  weights  are  established  at  the  beginning  of 
the  rating  cycle.  As  the  employee  works  to  accomplish  the  objectives,  the  supervisor  provides 
feedback  throughout  the  cycle  with  a  mandated  feedback  session  at  the  mid-point  of  the  cycle. 
The  objectives  may  be  either  equally  or  unequally  weighted.  Objectives  may  be  changed  up  to 
30  days  before  the  end  of  the  rating  cycle.  At  the  end  of  the  rating  cycle,  the  employee 
provides  narrative  comments  on  what  they  have  done  and  the  results  they  have  achieved 
towards  each  objective.  The  supervisor  considers  this  input,  adds  his  or  her  own  comments  as 
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part  of  the  rating  process,  and  uses  the  NSPS  performance  indicators  to  recommend  a  rating  for 
each  objective  using  this  five-point  scale: 

1  Unacceptable 

2  Fair 

3  Valued  Performer 

4  Exceeds  Expectations 

5  Role  Model 

The  supervisor  then  determines  the  contributing  factors'  influence  on  the  associated  job 
objective  and,  considering  the  standard  benchmarks,  indicates  whether  they  increase  the 
objective  rating  by  a  factor  of  1,  decrease  the  rating  by  a  factor  of  1,  or  have  no  effect,  0.  The 
adjustment  cannot  increase  a  rating  of  5,  reduce  a  rating  of  2,  or  change  a  rating  of  1.  In 
addition  to  these  rules  on  the  adjustment  of  individual  objective  ratings,  a  rating  of  1  on  any 


objective  will  result  in  an  overall  rating  of  1.  If  a  1 1 

objective  ratings  are  2  or  higher,  then  the 
appropriate  average  of  the  adjusted  scores 
(weighted  or  un-weighted)  is  computed  and 
rounded  to  a  final  score  using  the  rules  such  as 
those  in  Table  1.3. 


Table  1.3  -  Rules  for  Computing  Rating  of  Record 


Computed  Score 

Rating  of  Record 

1  on  any  objective 

1 

2.00  to  2.50 

2 

2.51  to  3.50 

3 

3.51  to  4.50 

4 

4.51  to  5.00 

5 

Ratings  recommended  by  a  supervisor  are 

reviewed  by  a  higher-level  reviewer  and  by  one  or  more  panels  of  managers  to  ensure 
consistency  and  fairness  across  the  pay  pool.  A  panel  may  represent  a  portion  of  the  pay  pool 
such  as  a  branch,  a  sub-pool,  or  the  entire  pay  pool.  In  all  cases  the  pay  pool  panel  reviews 
recommendations  to  ensure  consistency  and  fairness. 

Table  1.4  shows  the  link  between  ratings  and  payouts.  Shares  may  be  paid  as  salary  increases, 
bonuses,  or  both.  Employees  who  receive  a  rating  of  1  receive  no  pay  increase.  Those  receiving 
a  2  receive  the  pay  band  adjustment  (also  referred  to  as  Rate  Range  Adjustment),  which  is 
based  on  the  cost  of  labor  increase,  plus  the  local  market  supplement  (LMS).  Those  receiving  a 
3  or  higher  get  the  pay  band  adjustment  plus  the  LMS  and  an  incentive  increase  based  on  the 
number  of  shares  assigned  to  the  employee.  The  share  value  is  computed  as  a  percentage  and 
is  based  on  the  funding  established  by  the  pay  pool.  This  funding  includes  three  components. 
Element  1  includes  the  within-grade  increases  and  promotions  between  the  GS  system  grades 
that  were  banded  together.  Element  2  includes  any  portion  of  the  federal  government-wide 
cost  of  labor  increase  not  included  in  the  pay  band  adjustment.  Element  3  includes 
performance-based  bonuses  similar  to  performance  awards  in  the  GS  system).  Share  value  is 
overall  funding  (sum  of  Elements  1,  2,  and  3)  divided  by  the  sum  of  the  product  of  each 
employee's  base  pay  times  the  number  of  shares  he  or  she  received.  For  the  Spiral  1.1  payouts 
in  the  first  cycle,  none  of  the  cost  of  labor 

increase  was  dedicated  to  Element  2;  and  Table  1.4  -  Relationship  of  Rating  and  Payout 

all  pay  pools  had  a  value  of  0  percent  for 
Element  2  funding.  For  the  second  cycle, 

40  percent  of  the  cost  of  labor  increase 
was  dedicated  to  Element  2,  and  as  a 
consequence,  all  pay  pools  had  a  value  of 
1.0  percent  for  Element  2  funding.  *  LMS  =  Local  Market  Supplement,  if  applicable 

The  employee's  payout  is  computed  as  the  number  of  shares  the  employee  received  times  the 
share  value  times  the  employee's  base  pay.  This  payout  can  be  split  into  base  pay  increase  and 


Rating  (Score) 

Rating  of  Record 

1 

None 

2 

Rate  Range  Adjustment  (RRA)  +  LMS* 

3 

RRA  +  1  or  2  Shares  +  LMS 

4 

RRA  +  3  or  4  Shares  +  LMS 

5 

RRA  +  5  or  6  Shares  +  LMS 

1-8 


May  15,  2009 


NSPS  Spiral  1  Evaluation  Report 


1.  Introduction 


bonus.  The  basic  pay  increase  for  an  employee  cannot  take  them  above  the  top  of  the  pay  band 
and  may  be  subject  to  control  points  and  pro-ration. 

1.4.  Evaluation  Data  Sources 

This  NSPS  report  evaluation  report  analyzes  data  from  several  sources.  Table  1.5  provides  an 
overview  of  the  data  sources  used  to  measure  the  KPPs  and  their  associated  objectives. 

1.4.1.  Automated  Data 

Chapter  2  presents  analyses  based  on  data  from  automated  sources.  The  Performance 
Appraisal  Application  (PAA),  which  is  integrated  into  the  Defense  Civilian  Personnel  Data 
System  (DCPDS),  contains  performance  appraisal  plans  and  ratings  from  which  data  are 
uploaded  to  the  Compensation  Workbench  (CWB).  The  pay  pool  panel  uses  the  CWB  in  its 
process  of  assigning  final  ratings,  setting  the  number  of  shares,  allocating  payouts  between  pay 
increases  and  bonuses,  and  ensuring  that  the  total  payout  for  the  pay  pool  does  not  exceed  its 
funding.  Data  from  the  PAA  and  CWB,  supplemented  by  other  personnel  data  from  DCPDS, 
support  the  analysis  of  performance  ratings,  shares,  pay  increases,  and  bonuses. 

Extracts  from  DCPDS  also  form  the  basis  for  an  analysis  of  migration  within  NSPS,  migration  to 
non-NSPS  positions,  and  separation  from  DoD.  Chapter  2  also  examines  data  on  NSPS  requests 
for  reconsideration  and  on  Equal  Employment  Opportunity  (EEO)  complaints  from  both  NSPS 
and  non-NSPS  sources. 

The  Program  Executive  Office  (PEO)  for  NSPS  requested  a  study  of  performance  plans 
developed  under  NSPS.  Supervisors  use  performance  plans  to  document  the  objectives  that 
guide  employee  performance  during  the  performance  period,  rate  employee  performance 
relative  to  these  objectives,  and  justify  these  ratings.  Employees  contribute  their  self- 
assessments  to  the  performance  plans.  Chapter  2  also  presents  the  findings  from  this  study. 

1.4.2.  Site  Visits 

The  study  team  conducted  58  focus  groups  and  53  interviews  at  12  sites  and  reviewed 
documentation  relating  to  their  NSPS  implementation.  Focus  groups  typically  consisted  of  six 
to  12  participants;  however,  select  focus  groups  were  conducted  with  as  few  as  three  and  as 
many  as  20  participants.  Interviews  were  typically  conducted  with  one  to  three  participants. 

At  each  site,  separate  sessions  were  held  for  the  following: 

■  Employee  focus  groups 

■  Supervisor  focus  groups 

■  Pay  Pool  Managers/Pay  Pool  Panel  Members  group  interviews 

■  Implementation  Team  group  interview 

■  Commander/Director  interview 

■  Performance  Review  Authority  (PRA)  interview 

Interview  protocols  designed  for  each  group  elicited  responses  about  specific  aspects  of  NSPS. 
At  the  conclusion  of  each  focus  group  or  interview,  the  content  of  each  transcript  was  analyzed 
to  identify  salient  themes  and  specific  quotations  that  supported  the  themes.  Themes  were 
identified  using  the  following  criteria: 

■  The  prevalence  of  the  concept  in  the  transcript 

■  The  importance  of  the  theme  to  the  overall  research  question 

■  Internal  homogeneity  -  does  the  theme  express  a  unified  idea? 

■  External  heterogeneity  -  is  the  theme  unique  from  other  themes? 
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Table  1.5  -  Key  Performance  Parameters  Measured  by  Each  Data  Source 


Existing  Data 

Site  Visits 

Key  Performance  Parameter 

Personnel 

Data 

Performance 

Plans 

Attitude 

Survey 

Documents 

Employee 

Supervisor 

Ray  Pool 
Panel 

Implemen¬ 

tation 

Team 

High  Performing  Workforce  and  Management 

X 

X 

X 

X 

X 

X 

X 

X 

Stronger  link  between  total  compensation  and  performance 

X 

X 

X 

X 

X 

X 

Market  Sensitive  Salary  and  Wages 

X 

X 

X 

Performance  System  Provides  Ongoing  Feedback 

X 

X 

X 

X 

X 

X 

Stronger  link  between  retention  /  separation  and  performance 

X 

X 

X 

X 

X 

Performance  expectations  aligned  with  Department' sand 
organization's  strategic  plans,  mission  requirements 

X 

X 

X 

X 

X 

System  supports  employee  engagement  in  a  positive  manner 

X 

X 

X 

X 

Agile  and  Responsive  Workforce  and  Management 

X 

X 

X 

X 

Renewable,  expandable,  retractable  (size)  -  Mission 

Critical/Core  Support  Occupations 

X 

X 

Assignable,  deployable  (shape) 

X 

X 

X 

X 

Diverse  (multifaceted) 

Credible  and  Trusted  System 

X 

X 

X 

X 

X 

X 

X 

All  elements  of  NSPS  design  are  accessible  and  understandable 
(transparency) 

X 

X 

X 

X 

X 

X 

Management  decisionsare  merit-based,  n  on-discriminatory, 
and  consistent  with  NSPS  regulations 

X 

X 

X 

X 

X 

X 

Expeditious  and  fai  r  resolution  of  grievances 
(negotiated/admin),  performance  reconsiderations, 
classification  appeals,  adverse  action  appeals 

X 

X 

X 

X 

X 

Handle  performance  deficiencies  and  misconduct  timely  and 
deasively 

X 

X 

X 

X 

X 

NSPS  employees  in  aggregate  are  not  disadvantaged  by  the 
compensation  system 

X 

X 

X 

X 

X 

X 

Fiscally  Sound  System 

X 

X 

System  provides  for  cost  discipline  within  the  Department  top 
line 

X 

X 

Supporting  Infrastructure 

X 

X 

X 

X 

X 

X 

Employees  and  managers  are  knowledgeable  in  pertinent 
aspects  of  NSPS  as  a  result  of  training 

X 

X 

X 

X 

X 

X 

The  analysis  of  site  visit  data  resulted  in  the  identification  of  approximately  4,000  individual 
themes.  The  study  team  subsequently  reviewed  these  individual  themes  to  identify  a  reduced 
set  of  approximately  200  non-redundant  themes  for  each  group.  Finally,  the  study  team 
reviewed  these  themes  and  identified  a  reduced  set  of  16  findings  that  cut  across  groups. 
Chapter  3  presents  these  16  findings,  which  are  the  strongest  and  most  salient  findings  from 
across  all  sites  and  groups. 

1.4.3.  Attitude  Survey 

The  NSPS  Evaluation  Plan  identifies  the  Status  of  Forces  Survey-Civilian  (SOFS-C)  as  the  primary 
source  of  employee  attitudinal  data  for  use  in  evaluating  NSPS.  The  SOFS-C  is  a  Web-based 
survey  administered  by  the  Defense  Manpower  Data  Center  (DMDC)  to  a  sample  of  DoD  civilian 
employees.  It  measures  opinions  on  job  satisfaction,  quality  of  work  life,  and  performance 
management.  The  SOFS-C  has  included  several  items  on  awareness  of  and  attitudes  toward 
NSPS  since  October  2004.  The  SOFS-C  survey  was  moved  to  once  per  year  in  2007.  Areas  or 
topics  covered  by  the  survey  include: 

■  Background  data  on  the  respondent  (location,  personal  demographics,  supervisory  status, 
occupational  group,  conversion  spiral,  etc.) 
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■  Awareness  of  the  goals  or  mission  of  their  organization 

■  Attitudes  toward  the  work,  the  organization,  the  leaders,  and  leaders'  practices 

■  Supervisor  views  of  HR  management  tools,  authority,  processes,  and  quality 

■  Workforce  views  related  to  performance  and  retention 

Chapter  4  presents  select  results  that  are  relevant  to  an  evaluation  of  NSPS  from  the  latest 
SOFS-C  administration  as  well  as  longitudinal  data  for  those  questions  where  they  are  available. 

1.5.  Summary  Findings  of  the  Spiral  1  Evaluation 

The  data  sources  listed  above  provide  a  variety  of  perspectives  on  the  KPPs.  Chapter  5 
integrates  findings  across  sources  to  provide  a  broad  perspective  of  NSPS  Spiral  1 
implementation  relative  to  the  KPPs  and  additional  themes. 

1.6.  Appendices 

Seven  appendices  present  materials  to  supplement  the  evaluation  of  the  NSPS  Spiral  1 
implementation. 
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2.1.  Introduction  and  Method 

2.1.1.  Purpose 

This  chapter  summarizes  the  results  of  the  second  rating  and  payout  cycle  for  the  National 
Security  Personnel  System  (NSPS)  using  data  from  the  Performance  Appraisal  Application  (PAA) 
and  the  Compensation  Workbench  (CWB).  The  PAA  is  an  online  application  integrated  into  the 
Defense  Civilian  Personnel  Data  System  (DCPDS)  that  both  the  supervisor  and  employee  use  to 
propose  and  set  the  objectives  that  are  the  basis  for  assessment.  Using  the  PAA,  the  employee 
enters  his  or  her  accomplishments  for  the  rating  cycle  as  a  self-assessment  (not  mandatory,  but 
strongly  encouraged)  and  the  supervisor  then  completes  the  written  assessment  and 
recommended  ratings  based  on  those  objectives  and  the  applicable  contributing  factors.  An 
extract  from  the  PAA  is  then  created  for  each  pay  pool  and  downloaded  to  the  CWB,  an  offline 
application  developed  in  Visual  Basic  for  Applications  using  Microsoft  Excel.  The  CWB  assists  the 
Pay  Pool  Manager  and  his  or  her  team  in  the  process  of  assigning  final  ratings,  setting  the 
number  of  shares  and  allocating  the  payout  to  a  combination  of  base  pay  increase  and  bonus. 
The  CWB  computes  the  value  of  a  share  and  provides  tools  to  ensure  that  the  total  payout  for 
the  pay  pool  does  not  exceed  the  pay  pool  funding.  Once  the  Pay  Pool  Manager  certifies  the 
results  the  CWB  produces  an  output  file  that  is  then  uploaded  back  to  DCPDS  to  facilitate  the 
generation  of  the  appropriate  pay  transactions.  This  evaluation  uses  the  CWB  data  that  reflect 
the  NSPS  pay  pool  decision  process. 

Section  2.1  provides  an  overview  of  the  population  being  analyzed.  Section  2.2  describes 
performance  ratings  and  payout  results  from  the  pay  pool  process.  Section  2.3  covers  the 
equity  analysis  and  further  analyzes  rating  and  share  assignments.  Section  2.4  presents  the 
analysis  of  performance  plans.  The  final  section  examines  migration  to  and  from  NSPS. 

2.1.2.  Spiral  1  Population  Overview 

The  population  analyzed  during  this  second  NSPS  cycle  is  defined  as  the  group  of  individuals 
who  were  evaluated  and  had  their  compensation  determined  in  the  second  cycle  concluding 
September  30  or  October  31,  2007.  A  total  of  108,758  employees  were  included  in  the 
evaluation  files  produced  by  the  Spiral  1  organizations.  Table  2.1  provides  pay  pool  statistics  for 
the  population  by  component.  The  Army  had  the  largest  number  of  pay  pools  (402)  and  also 
the  largest  population.  The  Air  Force  had  the  smallest  pay  pool  (only  one  employee),  and  the 
Navy  the  largest  (2,465).  Since  the  evaluation  and  compensation  aspects  of  NSPS  are  the  major 
items  of  interest  in  this  chapter  we  removed  employees  from  this  population  who  were  not 
eligible  for  a  performance  rating  or  were  not  eligible  for  a  performance-based  salary  increase  or 
bonus.  Pay  pool  funding  was  based  on  107,979  employees,  of  whom  102,770  received  a  rating 
and  100,465  were  eligible  for  a  payout  during  the  second  cycle. 

Table  2.1  -  Pay  Pool  Size  Statistics 


Army 

402 

40,476 

4 

736 

100.8 

80.5 

78.0 

Navy 

121 

20,456 

4 

2,465 

169.1 

104.0 

253.2 

Air  Force 

369 

38,644 

1 

880 

104.8 

84.0 

96.7 

Fourth  Estate 

82 

9,182 

15 

422 

112.0 

81.0 

82.7 

Overall 

974 

108,758 

1 

2,465 

111.7 

84.5 

122.5 

Pay  Pool  Size  Distribution 
Minimum  I  Maximum  I  Mean  I  Median 


Standard  Deviation 


Component 


Number  of  Pay 
Pools 


Total 

Population 
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Table  2.2  provides  a  slightly  different  picture  of  Spiral  1.  It  displays  the  total  rated  population  by 
Pay  Schedule  and  Pay  Band.  Ninety-five  percent  of  the  rated  population  falls  within  the 
Standard  (72  percent)  and  Engineering  and  Scientific  (23  percent)  Career  Groups.  Of  the 
102,770  employees,  only  4,866  were  not  in  one  of  those  two  career  groups. 

Table  2.2  -  Distribution  by  Pay  Schedule  and  Pay  Band 


Pay  _ Pay  Band 


Schedule 

Band  1 

Band  2 

Band  3 

Band  4 

Total 

YA 

2,042 

34,272 

6,988 

43,302 

YB 

4,581 

5,641 

165 

10,387 

YC 

1,327 

13,626 

4,453 

19,406 

YD 

1,102 

11,955 

3,582 

16,639 

YE 

35 

373 

1,244 

224 

1,876 

YF 

71 

3,749 

1,776 

5,596 

YG 

164 

7 

171 

YH 

12 

1,549 

24 

1,585 

Yl 

214 

95 

4 

313 

YJ 

100 

224 

6 

17 

347 

YK 

94 

588 

94 

776 

YL 

12 

36 

31 

9 

88 

YM 

165 

47 

212 

YN 

898 

430 

46 

1,374 

YP 

698 

698 

Total 

11,351 

72,749 

18,420 

250 

102,770 

Section  2.4  will  provide  a  demographic  breakdown  of  Spiral  1  and  compare  the  results  with  the 
GS,  GM,  GG,  and  GL  employees  as  of  September  30,  2007. 

2.2.  Performance  Assessment 

This  section  provides  an  overview  of  performance  results  from  the  pay  pool  rating  process  and 
includes  discussions  of  objectives,  contributing  factors,  and  rating  and  share  distributions.  The 
charts  and  tables  below  are  based  on  the  100,465  employees  who  were  both  rated  and  payout 
eligible. 


2.2.1.  Performance  Objectives 

Supervisors  rated  their  employees  on 
one  to  ten  objectives  this  past  cycle. 
Table  2.3  shows  the  number  of 
employees  rated  on  any  given  number 
of  objectives.  Almost  50  percent  of 
employees  were  rated  on  the  basis  of 
three  objectives.  Ninety-three  percent 
of  all  employees  were  rated  on  the  basis 
of  three  to  five  objectives. 

In  addition  to  looking  at  the  number  of 
objectives  on  which  a  rating  was  based 


Table  2.3  -  Number  of  Objectives  Used  for  Rating 


Number  of 

Records  Using  This  Number 

Objectives  Used 

Number  of  Employees 

Percentage 

1 

576 

0.56% 

2 

4,299 

4.18% 

3 

49,903 

48.56% 

4 

33,312 

32.41% 

5 

10,405 

10.12% 

6 

1,431 

1.39% 

7 

398 

0.39% 

8 

113 

0.11% 

9 

20 

0.02% 

10 

8 

0.01% 

we  also  examined  the  use  of  weights  in  computing  the  actual  rating.  The  use  of  weights  allows 


the  supervisor  to  increase  or  decrease  the  emphasis  on  any  specific  objective.  The  default 


assumption  is  that  all  objectives  will  be  weighted  equally  in  determining  the  rating.  Table  2.4 


provides  summary  statistics  on  the  use  of  weights. 
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Table  2.4  -  Use  of  Weights  in  Determining  Ratings 


Since  the  default  assumption  is  that  all 
objectives  will  contribute  equally  to  the 
final  rating,  a  choice  to  use  equal 
weights  is  effectively  the  same  as  not 
assigning  weights  at  all.  In  effect,  about 
one  half  of  employees  had  extra 
emphasis  on  certain  objectives  by  using 
weights  while  the  other  half  had  equal 
emphasis  on  all  objectives.  Figure  2.1  shows  the  percentage  of  objectives  that  were  weighted 
by  component.  It  also  provides  a  table  of  the  more  popular  weighting  plans  and  the  frequency 
with  which  each  weighting  plan  was  used.  A  review  of  a  sampling  of  the  weighting  patterns 
does  not  indicate  a  correlation  between  the  final  rating  and  the  weighting  pattern.  The  analysis 
also  does  not  show  a  correlation  between  final  rating  and  number  of  objectives. 


Weighting  of 

Records  Using  This  Number 

Objectives 

Number  of  Employees 

Percent 

Not  Weighted 

45,927 

45.7% 

Equal  Weights 

3,811 

3.8% 

Unequal  Weights 

50,727 

50.5% 

Total 

100,465 

100.0% 

Popular  Weighting 
Patterns 

%  of  Weighted 
Plans 

40  /  30  /  30 

7.40% 

40  /  40  /  20 

6.10% 

50  /  30  /  20 

5.90% 

35  /  35  /  30 

4.60% 

50  /  25  /  25 

4.20% 

30  /  30  /  20  /  20 

4.00% 

40  /  35  /  25 

2.90% 

30  /  25  /  25  /  20 

2.90% 

40/  30/  20/  10 

2.80% 

40  /  20  /  20  /  20 

2.80% 

Figure  2.1  -Job  Objective  Weights  by  Component 


2.2.2.  Contributing  Factors 

The  above  discussion  of  the  number  and  weighting  of  objective  ratings  leads  logically  to  an 
analysis  of  the  actual  impact  of  the  contributing  factors  on  the  final  ratings.  After  individual 
objectives  have  been  evaluated,  the  supervisor  can  elect  to  increase  or  decrease  a  specific 
objective  rating  by  one  based  on  the  contributing  factors  identified  earlier.  Figure  2.2  shows  the 
impact  of  contributing  factors  on  each  objective  by  component.  A  change  in  an  objective  rating 
based  on  the  impact  of  contributing  factors  may  or  may  not  change  an  employee's  final  rating. 
Table  2.5  displays  the  relationship  of  overall  employee  ratings  both  before  and  after  the 
consideration  of  all  contributing  factors.  Both  sets  of  scores  use  the  same  weights. 


90% 

80% 

70% 

|  60% 

|  50% 

§■  40% 

^  30% 

20% 

10% 

0% 

Army  Navy  Air  Force  4th  Estate 


□  Decrease 
■  No  Change 

□  Increase 


Figure  2.2  -  Impact  of  Contributing  Factors  on  Each  Objective 
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The  cells  on  the  diagonal  in  Table  2.5  (i.e.,  those  shaded  in  blue)  represent  ratings  that  were 
unaffected  by  the  consideration  of  contributing  factors.  The  remaining  cells  with  non-zero 
entries  represent  instances  where  the  consideration  of  contributing  factors  resulted  in  a  change 
in  employee  ratings.  In  general,  the  consideration  of  contributing  factors  resulted  in  far  more 
increased  ratings  than  decreased  ratings  (17,581  increases,  388  decreases).  Almost  18  percent 
of  ratings  were  changed  as  the  result  of  the  contributing  factors.  According  to  NSPS  policy,  a 
rating  of  "1"  cannot  be  increased  by  contributing  factors  and  a  rating  of  "2"  cannot  be  lowered. 
Figure  2.3  presents  the  same  information  in  graphical  form  by  component  instead  of  rating.  The 
impact  is  consistent  across  all  components. 


Table  2.5  -  Impact  of  Contributing  Factors  on  Employee  Ratings 


Rating  After  Consideration  of  Contributing  Factors 


1 

2 

3 

4 

5 

Total 


224 

0 

0 

0 

0 

224 


2 

3 

4  5 

Total 

0 

0 

0 

0 

224 

1,128 

222 

0 

0 

1,350 

237 

53,263 

14,398 

0 

67,898 

0 

143 

25,127 

2,961 

28,231 

0 

0 

8 

2,754 

2,762 

1,365 

53,628 

39,533 

5,715 

100,465 

□  Decreased  Rating  ■  No  Effect  □  Increased  Rating 


Army  Navy  Air  Force  4th  Estate  Overall 


Figure  2.3  -  Impact  of  Contributing  Factors  on  Final  Rating  by  Component 


2.2.3.  Distribution  of  Ratings  and  Shares 

The  final  ratings,  as  modified  by  the  contributing  factors,  are  reviewed  at  various  levels  within 
the  pay  pools.  The  supervisory  hierarchy  reviews  and  discusses  each  rating  to  ensure  that  all 
employees  are  being  rated  fairly  and  consistently  within  the  pay  pool.  Supervisors  who 
routinely  give  higher  ratings  will  probably  find  their  recommended  ratings  lowered  unless  they 
have  evidence  to  support  that  higher  ratings  are  warranted.  Once  the  pay  pool  panel  agrees  on 
the  ratings  for  all  pay  pool  employees,  shares  are  awarded  in  accordance  with  Table  1.4.  The 
following  series  of  figures  provides  a  graphical  representation  of  the  distribution  of  both  ratings 
and  shares.  Figure  2.4  shows  the  overall  distribution  of  ratings  by  component.  Each  bar 
represents  the  percentage  of  employees  in  that  component  who  received  the  indicated  rating. 
Each  of  the  components  assigned  a  rating  of  3  to  more  than  57  percent  of  their  respective 
employees,  except  for  the  Fourth  Estate,  which  assigned  54  percent.  These  percentages 
correspond  with  the  population  counts  shown  in  Table  2.1. 
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A  high  percentage  of  employees  in  a  component  receiving  a  given  rating  will  necessarily  mean 
smaller  percentages  for  other  ratings.  Figure  2.4  and  Table  2.6  show  that  all  components  have 
very  similar  distributions.  Only  the  Fourth  Estate  stands  out,  with  three  percentage  points 
fewer  3  ratings  assigned  and  just  under  three  percentage  points  more  5  ratings  assigned. 


Distribution  of  Rating  by  Component 


Figure  2.4  -  Distribution  of  Ratings  by  Component 


Table  2.6  -  Rated  Population  Counts  by  Component 


Army 

90 

583 

21,063 

13,639 

1,575 

36,950 

Navy 

39 

342 

10,866 

6,941 

870 

19,058 

Air  Force 

76 

462 

20,510 

12,903 

1,997 

35,948 

Fourth  Estate 

19 

129 

4,589 

3,099 

673 

8,509 

Total 

224 

1,516 

57,028 

36,582 

5,115 

100,465 

Component 


The  pay  pools  have  flexibility  regarding  the  number  of  shares  to  award  based  on  an  employee's 
rating.  As  shown  in  Table  1.4,  the  pay  pool  can  assign  either  of  two  different  numbers  of  shares 
to  employees  with  a  rating  of  3  or  higher.  Consistent  with  business  rules  observed  during  the 
site  visits,  results  show  that  pay  pools  assigned  shares  based  upon  an  individual's  raw  score 
relative  to  the  range  of  scores  for  the  rating.  Of  those  individuals  with  a  rating  of  3,  98  percent 
with  a  raw  score  in  the  lower  half  of  the  range  (2.51  to  3.00)  received  one  share,  and  98 
percent  of  those  with  a  raw  score  in  the  upper  half  of  the  range  (3.01  to  3.50)  received  two 
shares.  Similarly,  of  those  individuals  with  a  rating  of  4,  97  percent  of  those  with  a  raw  score  in 
the  lower  half  of  the  range  (3.51  to  4.00)  received  three  shares  while  93  percent  of  those  with  a 
raw  score  in  the  upper  half  of  the  range  (4.01  to  4.50)  received  four  shares.  Of  those  individuals 
with  a  rating  of  5,  97%  with  a  raw  score  less  than  4.75  received  five  shares,  but  only  75%  of 
those  with  a  raw  score  above  4.75  received  six  shares. 

Figure  2.5  provides  a  display  of  the  distribution  of  shares  by  component  analogous  to  the 
distribution  of  ratings  displayed  in  Figure  2.4.  Since  more  than  56  percent  of  the  employees  in 
all  components  received  a  rating  of  3,  it  follows  from  Table  1.4  that  56  percent  or  more  of 
employees  would  receive  either  one  or  two  shares.  That  is,  the  percentage  receiving  either  one 
or  two  shares  will  be  the  same  as  the  percentage  receiving  a  rating  of  3.  For  those  receiving  a 
rating  of  3,  all  of  the  components  gave  two  shares  to  more  than  half  of  the  group.  The  most 
extreme  distribution  was  given  by  the  Fourth  Estate,  which  gave  two  shares  to  79  percent  of 
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the  employees  with  a  3  rating.  At  the  other  extreme,  the  Air  Force  gave  two  shares  to  67 
percent  of  employees  with  a  3  rating.  For  employees  with  a  rating  of  4,  most  of  them  received 
three  shares  rather  than  four.  The  Fourth  Estate  awarded  four  shares  to  the  most  employees 
(43  percent)  with  a  rating  of  4.  The  Army  awarded  four  shares  to  the  least  number  of 
employees  (26  percent)  with  a  rating  of  4.  All  of  the  components  awarded  five  shares  to  the 
majority  of  employees  who  received  a  rating  of  5.  The  percentages  in  Figure  2.5  correspond 
with  the  population  counts  shown  in  Table  2.7. 


Distribution  of  Shares  by  Component 


Figure  2.5  -  Distribution  of  Shares  by  Component 


Table  2.7  -  Share  Counts  by  Component 


Share  Distribution  (counts) 


Component 

0 

1 

2 

3 

4 

5 

6 

Total 

Army 

637 

5,276 

15,787 

10,056 

3,583 

1,074 

501 

36,950 

Navy 

381 

3,215 

7,651 

4,800 

2,141 

644 

226 

19,058 

Air  Force 

538 

6,701 

13,809 

8,655 

4,248 

1,276 

721 

35,948 

Fourth  Estate 

148 

967 

3,622 

1,782 

1,317 

443 

230 

8,509 

Total 

1,740 

16,159 

40,869 

25,293 

11,289 

3,437 

1,678 

100,465 

Awarding  a  higher  or  lower  number  of  shares  does  not  mean  that  an  organization  is  spending 
more  or  less  money.  The  total  amount  of  money  available  for  salary  increases  and  bonuses  is 
the  sum  of  Elements  1,  2,  and  3.  The  amount  received  by  a  given  employee  is  based  on  the 
product  of  the  number  of  shares  and  the  share  value  and  not  solely  on  the  number  of  shares 
received.  Awarding  a  larger  number  of  shares  will  reduce  the  value  of  each  share.  In  this  way 
the  system  is  self-correcting  and  the  pay  pool  controls  costs  through  the  choice  of  funding 
rates. 

Table  2.8  shows  the  distribution  of  the  number  of  shares  awarded  based  on  rating.  As  stated 
previously,  employees  who  receive  a  rating  of  1  or  2  do  not  receive  any  shares.  Since  224 
employees  received  a  rating  of  1  and  1,516  received  a  rating  of  2,  a  total  of  1,740  employees 
did  not  receive  any  shares.  Recall  that  for  each  rating  of  3  through  5  there  are  two  choices  for 
the  number  of  shares  awarded  (1  or  2  shares  for  a  rating  of  3,  3  or  4  shares  for  a  rating  of  4,  and 
5  or  6  shares  for  a  rating  of  5).  For  those  receiving  a  rating  of  3,  over  two-thirds  (72  percent) 
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received  the  higher  award  of  two  shares.  As  the  number  of  shares  increased  the  percentage 
receiving  the  higher  award  declined.  For  employees  with  a  rating  of  4,  only  a  little  less  than  one 
third  (31  percent)  received  the  higher  award  of  four  shares.  For  those  employees  with  the 
highest  rating  of  5,  about  a  third  (33  percent)  received  the  higher  award  of  six  shares. 


Table  2.8  -  Distribution  of  Shares  by  Rating 


1 

224 

0 

0 

0 

0 

0 

0 

224 

2 

1,516 

0 

0 

0 

0 

0 

0 

1,516 

3 

0 

16,159 

40,869 

0 

0 

0 

0 

57,028 

4 

0 

0 

0 

25,293 

11,289 

0 

0 

36,582 

5 

0 

0 

0 

0 

0 

3,437 

1,678 

5,115 

Total 

1,740 

16,159 

40,869 

25,293 

11,289 

3,437 

1,678 

100,465 

The  distribution  of  both  ratings  and  shares  can  be  displayed  across  a  variety  of  dimensions.  The 
distribution  by  pay  schedule  and  pay  band  will  be  of  particular  interest  to  some  readers.  In 
Spiral  1  all  fifteen  pay  schedules  were  represented.  Figure  2.6  is  an  aggregated  graphic  with  a 
pair  of  charts  for  each  of  the  fifteen  pay  schedules.  Each  pair  shows  the  distribution  of  ratings 
and  shares  by  pay  band  for  a  particular  pay  schedule,  with  the  ratings  distribution  on  the  left 
side  of  the  page  and  shares  distribution  on  the  right  side.  Since  different  pay  schedules  may 
have  different  numbers  of  pay  bands  the  actual  number  of  pay  bands  on  each  set  of  charts  may 
differ. 
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Figure  2.6-  Distribution  of 2007  Performance  Cycle  Ratings  and  Shares  by  Pay  Band  for  Each  Pay  Schedule 


80% 

60% 

40% 

20% 

0% 


■  Rating =2 
□  Rating  =  3 


Rating  by  Payband  for  Schedule  YA 


Shares  by  Payband  for  Schedule  YA 


Band  1 
0.4% 


Band  2 
0.3% 


Band  3 
0.2% 


17% 

61.2% 

33.3% 


0.9% 

42.0% 

472% 


3.5% 


9.7% 


80% 

60% 

40% 

20% 

0% 


□  Rating=1 

■  Rating  =2 

□  Rating=3 

□  Rating  =4 

■  Rating=5 


Rating  by  Payband  for  Schedule  YB 


Shares  by  Payband  for  Schedule  YB 


0.3% 

27% 

70.7% 


Band  2 


0.3% 

25% 

65.4% 


29.1% 

2.4% 


80%  -i 

60% 

40% 

o no/  _ 

a 

ZU/o 

AO/ 

1 1 

1  1 

U7o  - 

B 

and  1 

B 

and  2 

B 

and  3 

□  0  Shares 

3.1% 

2.8% 

0.6% 

■  1  Share 

27.9% 

22.3% 

10.3% 

□  2  Shares 

42.9% 

43.1% 

57.6% 

□  3  Shares 

18.2% 

21.3% 

21.8% 

■  4  Shares 

5.4% 

7.3% 

7.3% 

□  5  Shares 

1.5% 

19% 

1.2% 

S  6  Shares 

1.0% 

13% 

1.2% 

Rating  by  Payband  for  Schedule  YC 


Shares  by  Payband  for  Schedule  YC 


60% 

|— | 

40% 

20% 

pa/ 

1 

U/o 

Bandl 

Band  2 

Band  3 

□  Rating  =  1 

0.5% 

0.1% 

0.0% 

■  Rating  =2 

2.4% 

0.9% 

0.3% 

□  Rating =3 

67.2% 

47.8% 

27.4% 

□  Rating  =4 

27.4% 

44.5% 

53.2% 

■  Rating =5 

2.5% 

6.7% 

-B.f/o 

100% 

80% 

60% 

40% 

20% 

0% 

□  Rating  =  1 
■  Rating  =  2 


□  Rating  =  3 

□  Rating =4 
■  Rating  =  5 


Rating  by  Payband  for  Schedule  YD 


Band  1 
0.4% 
15% 


76.4% 

19.8% 

19% 


Band  2 
0.2% 
2.3% 


64.8% 

30.7% 

2.1% 


47.5% 

44.8% 


50% 

40%  i -  - - 

30% - - - - 

— 20%  I  _  n 

—  10% - 1  - 1  — 

0%  — ns® — — ®  1 -  iraH — — H 

_ =— I _ ■ ■_  Band  1  Band  2 

2  _ Band3 _  DO  Shares  _ 17% _ 24% _ 

3  0.0%  U  1  Share _ 37.9% _ 13.8% _ 

3  12%  □  2  Shares _ 38.6% _ 46.1% _ 


Shares  by  Payband  for  Schedule  YD 


Band  3 
12% 


2-8 


May  IS,  2009 


NSPS  Spiral  1  Evaluation  Report 


2.  Analysis  of  DCPDS/CWB  Data 


100% 

80% 

60% 

40% 

20% 

0% 

a  Rating =1 
B  Rating =2 

□  Rating =3 

□  Rating  =4 
B  Rating =5 


60% 

40% 

20% 

0% 

m  Rating =1 


B  Rating =2 
□  Rating =3 


□  Rating  =4 
B  Rating =5 


Rating  by  Payband  for  Schedule  YE 


-1 

1 

1: 

— 

Bandl 

2.9% 

0.0% 

829% 

14.3% 

0.0% 


Band  2 
0.5% 
2.7% 
69.7% 
26.5% 
0.5% 


Band  3 
0.0% 
17% 
69.5% 
27.3% 
15% 


Band  4 
0.0% 
0.0% 
54.0% 
42.4% 
3.6% 


Rating  by  Payband  for  Schedule  YF 


Band  2 
02% 


0.0% 

54.9% 


0.8% 

47.6% 


40.8% 

2.8% 


46.3% 

52% 


Band  3 
0.0% 


0.5% 

28.8% 


56.8% 
44.0 % 


Shares  by  Payband  for  Schedule  YE 


Shares  by  Payband  for  Schedule  YF 


60% 


40% 


20% 


0% 

□  Rating  =  1 
■  Rating =2 

□  Rating =3 

□  Rating  =4 


l  Rating  =5 


100% 

80% 

60% 

40% 

20% 

0% 


a  Rating =1 


■  Rating  =2 


□  Rating  =3 

□  Rating  =4 
B  Rating=5 


Rating  by  Payband  for  Schedule  YG 


40% 


Shares  by  Payband  for  Schedule  YG 


Bandl 

0.0% 


0.0% 

0.0% 


Band  2 
0.0% 
12% 
52.4% 
39.0% 


a 


Band  3 
0.0% 


28.6% 

28.6% 


7.3% 


Rating  by  Payband  for  Schedule  YH 


Shares  by  Payband  for  Schedule  YH 


o.o% 


0.0% 


83.3% 

16.7% 

0.0% 


36.5% 

3.1% 


45.8% 

37.5% 

*6.7% 


80% 


May  15 ,  2009 


2-9 


2.  Analysis  of  DCPDS/CWB  Data 


NSPS  Spiral  1  Evaluation  Report 


Rating  by  Payband  for  Schedule  Yl 


OVJ  /o 

60% 

40% 

20% 

no/ 

r 

U  70 

Band  1 

Band  2 

Band  3 

□  Rating  =  1 

0.5% 

0.0% 

0.0% 

■  Rating =2 

14% 

42% 

0.0% 

□  Rating  =  3 

710% 

52.6% 

0.0% 

□  Rating =4 

262% 

40.0% 

75.0% 

■  Rating  =  5 

0.9% 

3.2% 

25.0% 

Shares  by  Payband  for  Schedule  Yl 


Shares  by  Payband  for  Schedule  Yl 


80%  n 

60% 

Af)°/n  - 

*+u  /o 

9  no/ 

h 

n 

n 

ZU  /o  _ 

no/ 

1 

t: 

U  /o  - 

Band  1 

Band  2 

Band  3 

□  0  Shares 

19% 

4.2% 

0.0% 

■  IShare 

44.4% 

20.0% 

0.0% 

□  2  Shares 

26.6% 

32.6% 

0.0% 

□  3  Shares 

20.1% 

316% 

75.0% 

■  4  Shares 

6.1% 

8.4% 

0.0% 

□  5  Shares 

0.9% 

2.1% 

25.0% 

£1 6  Shares 

0.0% 

11% 

0.0% 

Shares  by  Payband  for  Schedule  Yl 


Rating  by  Payband  for  Schedule  YK 


80%  -i 

60% 

40% 

20% 

0% 

In  m 

Band  1 

Band  2 

Band  3 

□  0  Shares 

19% 

4.2% 

0.0% 

■  IShare 

44.4% 

20.0% 

0.0% 

□  2  Shares 

26.6% 

32.6% 

0.0% 

□  3  Shares 

20.1% 

316% 

75.0% 

■  4  Shares  6.1%  8.4%  0.0% 

IS  5  Shares 

0.9% 

2.1% 

25.0% 

■  6  Shares 

0.0% 

11% 

0.0% 

Shares  by  Payband  for  Schedule  YK 


50% 

40% 

30% 

20% 

10% 

0% 

□  0  Shares 
■  1  Share 

□  2Shares 


□  3  Shares 
■  4  Shares 
0  5  Shares 


Rating  by  Payband  for  Schedule  YL 

80% 


Shares  by  Payband  for  Schedule  YL 

80%  t — 


60% 

40% 

20% 


60% 

40% 

20% 

0% 


n 


JJ 

Band  1 


u/o 

Bandl 

Band  2 

Band  3 

Band  4 

□  Rating  =  1 

0.0% 

0.0% 

0.0% 

0.0% 

■  Rating =2 

8.3% 

0.0% 

0.0% 

0.0% 

□  Rating =3 

66.7% 

63.9% 

67.7% 

44.4% 

□  Rating =4 

25.0% 

30.6% 

32.3% 

55.6% 

■  Rating =5 

0.0% 

5.6% 

0.0% 

0.0% 

□  0  Shares 

■  IShare 

□  2Shares 

□  3Shares 

■  4  Shares 

□  5  Shares 
B  6  Shares 


8.3% 

0.0% 

66.7% 

13.7% 

8.3% 

0.0% 

0.0% 


0.0% 

-9.4% 

44.4% 

25.0% 

5.6% 

0.0% 

5.6% 


0.0% 

16.1% 

516% 

25.8% 

6.5% 

0.0% 

0.0% 


0.0% 

Itf/o 

33.3% 

44.4% 

111% 

0.0% 

0.0% 


2-10 


May  IS,  2009 


NSPS  Spiral  1  Evaluation  Report 


2.  Analysis  of  DCPDS/CWB  Data 


100% 

80% 

60% 

40% 

20% 

0% 


1  Rating =1 
■  Rating  =2 


□  Rating  =3 

□  Rating  =4 


Rating  by  Payband  for  Schedule  YP 


Shares  by  Payband  for  Schedule  YP 


o.o% 

32% 


77.5% 

17.0% 


50% 

40% 

30% 

20% 

10% 

0% 

□  0  Shares 

■  1  Share 

□  2  Shares 

□  3  Shares 

■  4  Shares 

□  5  Shares 


1 6  Shares 


2.2.4.  Compensation  (Payout)  Determination 

Earlier  in  this  chapter  we  discussed  the  concept  of  funding  the  pay  pools.  Element  2  funding, 
which  had  been  zero  for  the  first  cycle,  was  set  at  one  percent  for  all  pay  pools  for  the  second 
cycle.  Element  1  and  Element  3  funding  percentages  varied  across  pay  pools.  Depending  on 
decisions  made  by  the  individual  components  and  organizations,  the  flexibility  that  the  pay  pool 
has  to  set  these  funding  percentages  differed.  For  this  cycle  there  were  974  pay  pools  belonging 
to  the  four  components.  Table  2.9  displays  the  distribution  of  Element  1,  Element  3,  and  total 
funding  across  the  pay  pools,  grouped  by  component.  We  chose  to  show  statistics  in  this 
manner  rather  than  show  the  individual  values  for  all  974  pay  pools. 


Table  2.9  -  Funding  Percentage  Statistics  by  Component 


Component 

Number  of 
Pay  Pools 

Funding 

Element 

Mean 

Mode 

Standard 

Deviation 

Minimum 

Maximum 

Army 

402 

Element  1 

2.51 

2.26 

0.476 

0.00 

4.64 

Element  3 

2.02 

1.50 

0.854 

0.00 

5.69 

Navy 

121 

Element  1 

2.19 

2.26 

0.527 

1.10 

3.47 

Element  3 

1.11 

1.00 

0.527 

0.00 

4.00 

Air  Force 

369 

Element  1 

2.32 

2.26 

0.171 

1.02 

2.89 

Element  3 

1.47 

1.00 

0.564 

0.50 

6.25 

Fourth  Estate 

82 

Element  1 

2.37 

2.26 

0.342 

0.00 

3.00 

Element  3 

2.30 

2.50 

0.828 

0.71 

4.69 

Total 

974 

Element  1 

2.39 

2.26 

0.402 

0.00 

4.64 

Element  3 

1.72 

1.50 

0.805 

0.00 

6.25 

The  Army  and  Fourth  Estate  had  one  pay  pool  each  with  Element  1  funding  set  to  zero.  The 
Army  and  the  Navy  had  one  pay  pool  each  with  Element  3  funding  set  to  zero.  The  most 
common  Element  1  funding  level  across  all  components  was  2.26  percent.  The  most  common 
Element  3  funding  level  was  1.0  percent  in  the  Air  Force  and  Navy,  1.5  percent  in  the  Army,  and 
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2.5  percent  in  the  Fourth  Estate.  The  Navy  had  the  greatest  variation  among  Element  1  funding 
levels  with  a  standard  deviation  of  0.527,  while  the  Air  Force  had  the  least  variation  with  a 
standard  deviation  of  0.171.  The  Army  had  the  greatest  variation  among  Element  3  funding 
levels  with  a  standard  deviation  of  0.854,  while  the  Navy  had  the  smallest  variation  with  a 
standard  deviation  of  0.527.  An  Army  pay  pool  had  the  highest  Element  1  funding  level  at  4.64 
percent,  while  the  Air  Force  had  the  highest  Element  3  funding  level  at  6.25  percent. 

2.2.5.  Share  Values 

The  actual  determination  of  payout  amounts  is  based  on  the  concept  of  a  share  value.  Share 
value  is  calculated  by  dividing  the  pay  pool  fund,  expressed  in  dollars,  by  the  summation  of 
each  employee's  base  salary  on  the  last  day  of  the  appraisal  period  times  the  number  of  shares 
earned  by  the  employee. 

Share  Value  =  Pay  Pool  Fund($)  /  Z  (base  sal  ary  employee  x  sharesempioyee) 

Share  value  is  expressed  as  a  percentage  of  base  salary.  All  employees  in  a  pay  pool  will  have 
the  same  share  value. 

Since  the  total  payout  across  the  pay  pool  cannot  exceed  the  pay  pool  funding,  then  the  sum  of 
the  Elements  1,  2,  and  3  funding  must  equal  the  sum  of  the  payouts  received  by  all  employees 
in  the  pay  pool. 

An  employee's  performance  payout  is  calculated  by  multiplying  the  employee's  base  salary  at 
the  end  of  the  appraisal  period  by  the  number  of  shares  earned  by  the  employee  by  share 
value. 


Employee  Performance  Payout  =  Base  Salary  x  Shares  x  Share  Value 

In  effect,  the  share  value  adjusts  the  payouts  to  ensure  that  the  overall  funding  is  not  exceeded. 
If  a  pay  pool  is  very  generous  in  awarding  ratings  and  shares,  then  the  share  value  will  decrease 
to  keep  the  expenditure  in  line  with  funding.  Conversely,  if  the  pay  pool  awards  fewer  shares, 
then  the  share  value  will  increase. 

Table  2.10  shows  how  the  share  values  were  distributed  across  the  components.  The  share 
value  serves  as  the  buffer  that  prevents  overspending  while  maintaining  the  relative  payouts 
determined  by  the  ratings  and  share  distributions. 


Table  2.10  -  Share  Value  Distribution  by  Component 


Component 

Number  of 
Pay  Pools 

Mean 

Standard 

Deviation 

Minimum 

Maximum 

Army 

402 

2.36 

0.579 

0.91 

6.11 

Navy 

121 

1.84 

0.580 

0.84 

3.81 

Air  Force 

369 

2.00 

0.410 

1.11 

3.80 

Fourth  Estate 

82 

2.20 

0.407 

1.37 

3.61 

Total 

974 

2.15 

0.544 

0.84 

6.11 

2.2.6.  Salary  Increases  and  Bonuses 

Table  2.11  below  provides  abbreviated  summary  statistics  concerning  payouts.  In  aggregate, 
the  pay  pools  paid  out  3.1  percent  less  in  salary  increases  than  the  combined  funded  amount 
from  Element  1  and  Element  2.  Rather  than  being  an  indication  of  intentionally  withheld 
payouts,  a  major  portion  of  this  difference  reflects  the  fact  that  pay  pool  funding  is  based  on 
both  rated  individuals,  eligible  for  a  performance  payout,  as  well  as  non-rated  individuals,  not 
eligible  for  a  performance  payout.  In  addition,  NSPS  rules  allow  for  Element  1  funds  to  be  used 
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for  bonuses,  sometimes  a  necessity  when  employees  hit  the  end  of  their  pay  range  or  a  control 
point  for  their  position.  This  rollover  resulted  in  bonus  payouts  5.5  percent  greater  than 
Element  3  funding.  Total  payouts  were,  in  aggregate,  two-tenths  of  one  percent  lower  than  the 
total  Element  1,  Element  2,  and  Element  3  funding. 


Table  2.11  -  Payout  Summary  Figures 


Summary  Statistics 

Maximum  payout  (percent) 

Maximum  payout  (value  in  dollars) 

Number  not  receiving  rate  range  adjustment 

Value  in  dollars  withheld  from  rate  range  adjustments 
Number  hitting  top  of  band 

Percent  of  population  hitting  top  of  band 

23.2% 

$27,733 

243 

$214,768 

4,206 

4.1% 

Element  1  and  2  -  Spent 

Funded 

Percent  Spent 

$236,935,144 

$244,446,317 

96.9% 

Element  3  -  Spent 

Funded 

Percent  Spent 

$127,624,047 

$120,999,175 

105.5% 

Total  Funding  -  Spent 

Funded 

Percent  Spent 

$364,559,191 

$365,445,492 

99.8% 

Table  2.12  indicates  the  number  and  percent  of  employees  receiving  salary  increases  and 
bonuses.1  A  total  of  1,740  employees  (about  2  percent  of  the  group)  received  neither  a  salary 
increase  nor  a  bonus.  As  stated  previously,  these  employees  received  no  shares  and  thus  could 
not  receive  a  performance-based  payout.  About  86  percent  of  the  group  received  both  a 
performance  salary  increase  and  a  performance  bonus.  The  remaining  12  percent  of  the 
employee  population  was  split  between  those  who  received  a  bonus  but  no  salary  increase  (3 
percent)  and  those  who  received  a  salary  increase  but  no  bonus  (9  percent). 


Table  2.12  -  Payout  Summary  Statistics 


Table  2.13  shows  the  average  payout  by  rating,  and  Table  2.14  provides  the  average  payout  by 
base  salary.  The  tables  list  the  average  dollar  amount  and  average  percentage  increase  in 
salary,  bonus,  and  total  payout.  The  percentages  listed  in  these  tables  are  weighted  averages 
based  on  base  salary.  While  an  increase  in  percentage  payout  is  expected  as  rating  increases, 
the  very  pronounced  increase  in  percentage  payout  as  base  salary  increases  reflects  a  concern 
that  higher  paid  employees  receive  higher  ratings. 


1  _ 

The  analysis  of  ratings  and  shares  was  based  on  the  population  that  received  a  rating  and  share  allocation.  The  payout 

analysis  excludes  2,304  employees  who,  though  rating-eligible,  were  not  payout-eligible.  We  believe  the  majority  of  those 

listed  as  not  payout-eligible  were  rated  in  one  pay  pool  and  then  moved  and  were  funded  by  another  pay  pool. 
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Table  2.13  -  Average  Payout  by  Rating 


Rating 

Salary  Increase 

Bonus 

Payout 

Dollar  value 

Percent 

Dollar  value 

Percent 

Dollar  value 

Percent 

1 

$0 

0% 

$0 

0% 

$0 

0% 

2 

$0 

0% 

$0 

0% 

$0 

0% 

3 

$1,583 

2.46% 

$848 

1.27% 

$2,431 

3.73% 

4 

$3,263 

4.53% 

$1,727 

2.28% 

$4,990 

6.81% 

5 

$5,333 

6.66% 

$3,139 

3.66% 

$8,472 

10.32% 

Average 

$2,358 

3.39% 

$1,270 

1.74% 

$3,628 

5.13% 

Table  2.14  -  Average  Payout  by  Base  Salary 


Rating 

Salary  Increase 

Bonu 

s 

Payout 

Dollar  value 

Percent 

Dollar  value 

Percent 

Dollar  value 

Percent 

Below  40K 

$977 

2.93% 

$504 

1.49% 

$1,481 

4.42% 

40-60 K 

$1,714 

3.35% 

$749 

1.48% 

$2,463 

4.83% 

60-80 K 

$2,456 

3.52% 

$1,048 

1.50% 

$3,504 

5.02% 

80-100K 

$2,949 

3.33% 

$1,866 

2.12% 

$4,815 

5.45% 

100K+ 

$4,164 

3.78% 

$3,035 

2.70% 

$7,199 

6.48% 

Average 

$2,358 

3.39% 

$1,270 

1.74% 

$3,629 

5.13% 

2.2.7.  Net  Draw 

Because  the  total  sum  available  for  salary  increases  and  bonuses  is  effectively  determined  by 
setting  aside  a  percentage  of  each  individual's  base  pay,  then  each  individual  can  be  viewed  as 
having  that  "factor"  in  the  pay  pool.  An  obvious  question  is,  "How  much  did  each  individual 
receive  in  payouts  relative  to  his  or  her  factor?"  To  answer  this  question,  we  calculated  a  value 
called  net  draw,  which  is  equal  to  the  amount  an  individual  received  in  payouts  minus  the 
amount  of  the  individual's  factor  in  the  overall  funding.  A  positive  net  draw  would  indicate  that 
an  individual  received  more  in  payouts  than  his  or  her  factor.  Conversely,  a  negative  net  draw 
results  when  an  individual  receives  less  in  payouts  than  his  or  her  factor.  We  use  net  draw  to  be 
able  to  compare  groups  across  pay  pools  and  reduce  the  impact  of  different  funding  levels. 

All  individuals  in  a  pay  pool  generally  have  their  salaries  included  in  the  funding  calculations, 
but  not  all  employees  are  necessarily  eligible  for  ratings  and  payouts.  Individuals  who  have  less 
than  90  days  in  an  NSPS  position  at  the  time  of  the  rating  cycle  are  not  be  eligible  for  a  rating. 
In  those  cases  the  individuals  will  contribute  to  the  funding  of  the  pay  pool  but  will  not  be 
eligible  for  a  performance-based  salary  increase  or  bonus.  The  remaining  rated  individuals 
benefit  from  the  additional  funding  set  aside  from  the  non-rated  employees  in  the  pay  pool. 

As  non-rated  individuals  are  not  included  in  net  draw  computations,  overall  net  draw  is 
positive.  Thus,  individual  payouts  are  a  function  of  both  the  pay  pool  funding  rate  and  the 
number  of  non-rated  individuals  in  the  pay  pool.  In  general,  all  other  things  being  equal, 
individuals  in  pay  pools  with  a  higher  percentage  of  non-rated  individuals  will  receive  higher 
payouts. 

In  defining  the  payout  analysis  group,  we  included  only  individuals  who  were  eligible  for  ratings 
and  payouts.  Assuming  that  there  were  at  least  some  members  in  each  organization  whose 
salaries  were  included  in  the  funding  calculations  but  were  not  eligible  for  payouts,  the 
remaining  group  would  be  expected  to  have  an  average  net  draw  that  is  positive.  Figure  2.7 
shows  net  draw  percentage  for  each  pay  schedule.  The  position  of  the  circles  indicates  the 
percent  net  draw  while  the  area  of  the  circles  reflects  the  relative  size  of  the  population  in  the 
associated  pay  schedule.  Note  that  pay  schedules  YB,  YD,  YE,  YL,  YM,  YN,  and  YP  all  had 
negative  net  draw  percentages. 


2-14 


May  15,  2009 


NS  PS  Spiral  1  Evaluation  Report 


2.  Analysis  of  DCPDS/CWB  Data 


In  addition  to  looking  at  these  general  statistics  for  net  draw,  we  looked  at  the  distribution  of 
net  draw  across  a  variety  of  other  variables.  One  analysis  grouped  employees  into  five  pay 
ranges  based  on  base  pay  and  then  examined  the  average  net  draw  percentage  for  each  pay 
range.  Figure  2.8  shows  a  positive  relationship  between  pay  and  net  draw  percentage. 


Pay  Schedule 


Figure  2.7  -  Draw  by  Pay  Schedule 
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Figure  2.8  -  Net  Draw  Percentage  by  Salary  Range 


2.2.8.  Distribution  of  Ratings,  Shares,  and  Payouts  by  Pay  Ranges 

In  the  discussion  above  there  appears  to  be  a  relationship  between  net  draw  and  base  pay.  We 
carried  this  analysis  a  bit  further  and  looked  at  two  additional  relationships.  The  first 
relationship  we  examined  was  between  base  pay  and  the  rating  received.  The  second 
relationship  examined  whether  an  individual  received  the  higher  or  lower  possible  number  of 
shares.  Obviously  this  reduces  the  population  to  those  who  received  a  rating  of  three  or  higher. 
We  looked  at  the  relationship  between  the  number  of  shares  option  and  base  pay  for  this 
subset  of  the  population. 
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Because  rating  is  a  measure  of  the  degree  to  which  employees  met  their  objectives,  which 
reflect  the  level  of  work  required  by  their  positions,  we  would  expect  to  see  a  relatively  even 
distribution  of  ratings  across  all  pay  schedules,  pay  bands,  and  salary  ranges.  Each  employee 
should  have  the  same  opportunity  to  succeed  or  fail  based  upon  the  objectives  established. 
However,  Figure  2.9  shows  the  average  rating  increases  as  a  function  of  salary.  Those  in  the 
highest  pay  range  have  an  average  rating  of  3.8  while  those  in  the  lowest  pay  range  have  an 
average  rating  of  3.3.  This  result  is  consistent  with  the  chart  in  Table  2.14  which  shows  average 
payout  by  base  salary.  The  weighted  average  percent  payout  of  those  in  the  highest  salary 
range  is  more  than  two  percentage  points  higher  than  those  in  the  lowest  salary  range. 2 

The  second  relationship  of  concern  is  how  the  number  of  shares  options  were  exercised  relative 
to  base  pay.  The  data  addressing  this  question  are  somewhat  mixed.  Figure  2.10  shows  the 
percentage  of  employees  with  a  rating  of  3  who  received  two  shares  (vs.  one  share)  broken  out 
by  salary.  The  percentage  receiving  the  higher  share  value  shows  the  same  pattern  as  the 
average  rating  pattern  shown  in  Figure  2.9,  in  which  the  data  are  broken  out  by  salary  range. 
The  percent  receiving  two  shares  is  almost  twenty  percentage  points  higher  for  those  in  the 
highest  salary  range  compared  to  those  in  the  lowest.  The  same  pattern  also  exists  for  those 
with  a  rating  of  4  as  shown  in  Figure  2.11.  Of  those  in  the  highest  salary  range,  44%  received  4 
shares  (vs.  3  shares)  compared  to  only  23  percent  of  those  in  the  lowest  salary  range.  The 
pattern  is  reversed  for  those  with  a  5  rating  as  shown  in  Figure  2.12. 

Section  2.2.2  above  looked  at  the  impact  of  contributing  factors  on  final  rating.  Figure  2.3 
showed  the  percent  of  employees  who  had  their  final  rating  increased,  decreased,  or  remained 
unchanged,  by  component.  Given  the  empirical  correlations  between  base  salary  and  rating, 
and  base  salary  and  payout  reported  above,  we  also  looked  at  how  the  impact  of  contributing 
factors  on  final  rating  varies  by  base  pay.  Figure  2.13  shows  that  the  percent  of  employees 
whose  rating  was  increased  as  a  result  of  contributing  factors  increases  linearly  with  base  pay. 
There  is  almost  a  ten  percentage  point  difference  between  those  in  the  lowest  salary  range 
(11.7%)  and  those  in  the  highest  salary  range  (21.0%). 


Average  Rating 


Salary  Range 


Figure  2.9  - Average  Rating  by  Salary  Range 


2  Weighting  is  based  on  base  salary 
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Figure  2.10  -  Number  of  Shares  for  a  Rating  of  3  by  Salary  Range 
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Figure  2.11 -  Number  of  Shares  for  a  Rating  of  4  by  Salary  Range 
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Figure  2.12  -  Number  of  Shares  for  a  Rating  of  5  by  Salary  Range 
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Impact  of  Contributing  Factors 
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Figure  2,13  -  Impact  of  Contributing  Factors  on  Rating  by  Salary  Range 

Thus,  the  above  charts  indicate  that,  in  general,  the  higher  the  pay: 

■  The  higher  the  rating 

■  The  higher  the  proportion  getting  the  higher  number  of  shares  for  ratings  of  3  or  4 

■  The  higher  the  percent  who  received  an  increased  rating  due  to  contributing  factors 

■  The  higher  the  payout  percentage 

The  data  suggest  that  organizations  with  similar  results  should  reflect  on  whether  performance 
expectations  are  set  appropriately  and  whether  some  pay  schedule/pay  band  groups  are  more 
likely  to  meet  criteria  for  ratings  above  the  "valued  performer"  level.  Inappropriate 
performance  expectations  can  inadvertently  shift  pay  pool  dollars  from  the  lower  paid 
employees  to  the  higher  paid  ones.  Higher  pay  is  generally  consistent  with  higher  grade,  more 
experience,  and  a  greater  contribution  to  organizational  performance.  It  is  not  clear  whether 
higher  ratings  are  more  likely  to  be  assigned  to  employees  perceived  as  making  larger 
contributions  to  the  organization.  Managers  must  be  trained  not  to  rate  the  value  of  the 
position,  but  to  rate  performance  in  terms  of  the  more  demanding  objectives  associated  with 
these  positions. 

2.2.9.  Payout  Split  Strategies 

Splitting  an  employee's  performance  payout  between  a  salary  increase  and  a  bonus  can  have  a 
significant  impact  on  the  employee's  earnings,  both  for  the  current  cycle  and  in  the  future.  A 
salary  increase  can  generally  be  thought  of  as  "the  gift  that  keeps  on  giving"  because  the 
employee  will  continue  to  receive  this  payment  in  future  years,  and  it  increases  base  salary, 
from  which  future  increases  are  based.  A  bonus,  on  the  other  hand,  does  not  affect  future 
compensation. 

In  some  instances  there  is  really  no  decision  to  be  made.  If  an  employee  is  at  or  near  the 
maximum  for  his  or  her  pay  band,  then  the  portion  of  the  payout  that  can  be  given  as  a  salary 
increase  may  be  severely  limited  and  all  or  most  of  the  payout  may  have  to  be  in  the  form  of  a 
bonus.  Splits  may  also  be  affected  by  the  use  of  control  points  to  limit  salary  growth  as 
discussed  above.  Trying  to  categorize  salary  split  strategies  is  difficult  to  do  with  any  precision. 
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Although  the  calculation  of  a  salary  split  is  objective,  the  interpretation  of  the  strategy  behind 
the  salary  split  is  highly  subjective.  We  can  only  observe  the  results  and  then  try  to  see  if  there 


are  patterns.  Because  funding  for  Elements  1  and  2  generally  represents  salary  increases  and 
Element  3  generally  represents  bonus  funding,  we  might  expect  to  see  payout  splits  that 


approach  the  funding  splits.  Base  pay  increases  for  the  pay  pool  cannot  exceed  the  sum  of 


Element  1  and  Element  2  funding. 
Other  restrictions  are  1)  a  pay 
pool's  total  payout  is  limited  only 
by  the  total  funding  and,  2) 
individual  salary  increases  are 
limited  by  the  maximum  of  the  pay 
band  or  control  point.  Table  2.15 
below  shows  the  average  pay  pool 
percentage  split  between  salary 
payouts  and  bonuses,  both  funded 


Table  2.15  -  Payout  Distribution 


Funded* 

Allocated 

tompuneni 

Salary 

Bonus 

Salary 

Bonus 

Army 

63.5% 

36.5% 

62.8% 

37.2% 

Navy 

74.9% 

25.1% 

69.2% 

30.8% 

Air  Force 

70.3% 

29.7% 

68.7% 

31.3% 

Fourth  Estate 

58.7% 

41.3% 

56.8% 

43.2% 

Average 

66.9% 

33.1% 

65.0% 

35.0% 

*Proportion  of  Elements  1  and  2  to  total  pay  pool  funding  =  salary 


split 


and  actually  allocated,  by  component. 


2.2.10.  Use  of  Control  Points  in  Limiting  Pay 

NSPS  includes  the  concept  of  control  points  to  allow  organizations  to  manage  pay  progression 
for  similar  positions  in  a  pay  band.  Control  points  limit  pay  rates  and  pay  increases  and  slow 
movement  through  a  pay  band.  By  establishing  a  control  point  below  the  maximum  for  a  pay 
band,  managers  can  control  the  size  of  potential  increases  so  that  pay  does  not  grow  too  fast  or 
exceed  the  market  value  for  the  job.  The  CWB  software  allows  pay  pools  to  set  control  points 
and  then  enter  either  a  "Yes"  or  "No"  to  allow  a  control  point  to  be  exceeded.  The  increase  will 
be  limited  at  the  control  point  only  if  a  "No"  is  entered  to  this  question.  During  the  first  cycle 
there  was  some  indication  that  not  all  pay  pools  fully  understood  either  the  intent  of  the 
control  points  or  how  to  use  them.  One  Spiral  1.1  organization  used  them  deliberately  to 
distinguish  between  jobs  of  significantly  different  scope  in  the  same  pay  band.  Control  points 
were  used  much  more  extensively  in  the  second  cycle.  Table  2.16  shows  how  employees  were 
affected  by  control  points  across  the  components.  The  table  breaks  out  the  number  of 
employees  who  had  a  valid  control  point  value  in  the  CWB,  the  number  who  hit  that  control 
point,  and  the  number  who  had  their  salary  growth  limited  by  the  control  point. 


Table  2.16  -  Control  Point  Statistics 


Component 

Payout 

Eligible 

Employees 

Valid  Control  Point  in 
CWB 

(percent  of  eligible) 

Hit  Control  Point 
(percent  of  those  with 
CP) 

Stopped  at  Control 
Point  (percent  of  those 
that  hit  CP) 

Army 

37,121 

6,683  (18%) 

1,068  (16%) 

840  (79%) 

Navy 

19,058 

4,957  (26%) 

1,102  (22%) 

814  (74%) 

Air  Force 

35,779 

3,266  (9%) 

400  (12%) 

261  (65%) 

Fourth  Estate 

8,509 

2,040  (24%) 

298  (15%) 

248  (83%) 

Total 

100,467 

16,946  (17%) 

2,868  (17%) 

2,163  (75%) 

Appendix  A  provides  an  analysis  of  payouts  related  to  Extraordinary  Pay  Increases  (EPI), 
Organizational/  Team  Achievement  Recognition  (OAR),  and  the  use  of  pro-ration. 

2.3.  Equity  Analysis 

This  section  addresses  potential  effects  of  NSPS  on  subgroups  of  employees,  especially  the 
protected  subgroups  including  females,  minorities,  individuals  40  years  of  age  and  older,  and 
disabled  persons.  The  NSPS  pay  pool  process  was  established  to  foster  fairness  and  equity  for 
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all  employees.  Ratings  recommended  by  a  supervisor  are  reviewed  both  at  a  higher  level  and  by 
a  pay  pool  panel  -  a  group  of  management  officials  that  ensures  rating  and  payout  consistency 
and  fairness  across  the  employees  who  are  members  of  the  pay  pool.  In  larger  organizations 
reviews  also  may  be  made  at  an  intermediate,  sub-pay  pool  level  in  the  organization,  with  the 
pay  pool  panel  ensuring  consistency  and  fairness  across  sub-pay  pools.  Raters,  reviewers,  and 
panel  members  apply  standard,  NSPS-wide  performance  indicators  and  benchmarks  when  they 
consider  employees'  performance  assessments.  Thus,  the  review  process  at  the  pay  pool  level 
is  designed  to  ensure  that  the  standard  criteria  are  applied  fairly  to  the  performance  of  each 
individual  in  the  pay  pool,  without  regard  to  status  among  protected  subgroups.  In  addition, 
there  is  a  rating  reconsideration  process  for  employees  who  wish  to  challenge  their  ratings. 
Ensuring  such  consistency  of  performance  ratings  within  a  pay  pool  is  essential  for  fair 
performance-based  pay  within  a  pay  pool. 

The  operational  autonomy  of  the  pay  pool  fosters  equity  and  consistency  within  the  pay  pool 
and  typically  keeps  rating  decisions  at  a  level  close  to  the  employees  and  their  work.  It 
complicates  analysis  of  aggregate  measurement  across  the  pay  pools,  however.  Each  pay  pool  is 
unique,  with  size  and  composition  varying  widely.  Performance  expectations  vary  due  to  the 
diversity  of  jobs  and  experience  within  and  across  pay  pools  and  the  wide  range  of  work 
demands  included  in  some  pay  bands.  While  individual  ratings  are  based  on  performance 
compared  to  individual  objectives  and  performance  indicators,  challenging  jobs  are  more  likely 
than  routine  ones  to  present  opportunities  for  a  high  rating.  Any  given  pay  pool  may  have  a 
high  proportion  of  high  performers  or  norm  its  work  with  relatively  fine  distinctions  at 
successful  rating  levels.  For  these  reasons,  analyzing  rating  outcomes  across  pay  pools  is 
challenging,  and  analyzing  differences  in  subgroup  results  across  pay  pools  is  particularly 
problematic. 


Figure  2-14  below  demonstrates  the  difficulty.  The  figure  depicts  a  histogram  of  the  2007  pay 
pools  broken  out  by  the  average  rating  within  the  pay  pool.  The  wide  range  of  average  ratings 
across  pay  pools  illustrates  how  the  use  of  average  rating  as  a  measure  of  a  subgroup's 
performance  can  be  misleading.  If  a  subgroup  is  concentrated  in  pay  pools  with  lower  average 
ratings,  its  average  rating  across  all  pay  pools  probably  will  be  lower  than  the  average  rating  for 
everyone.  Thus,  it  is  possible  for  members  of  a  subgroup  to  receive  ratings  that  are  the  average 
within  their  respective  pay  pools,  but  the  subgroup's  aggregate  rating  could  be  below  the 
overall  average. 


Histogram  of  2007  Pay  Pool  Average  Ratings 


A® 


Pay  Pool  Average  Rating 


Figure  2.14  -  Histogram  of  Pay  Pool  Average  Ratings 
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Table  2.17-  Protected  Subgroups 


Race 

Population 

Percent 

White 

72,809 

72.5% 

Black/ African  American 

14,466 

14.4% 

Asian 

5,704 

5.7% 

American  Indian/Alaska  Native 

794 

0.8% 

Native  Hawaiian/Pacific  Islander 

561 

0.6% 

Multiracial 

715 

0.7% 

Identity  Pending 

253 

0.3% 

Unspecified 

5,163 

5.1% 

Total 

100,465 

100.0% 

Gender 

Population 

Percent 

Female 

40,749 

40.6% 

Male 

59,716 

59.4% 

Total 

100,465 

100.0% 

Ethnicity 

Population 

Percent 

Non-Hispanic 

94,293 

93.9% 

Hispanic 

5,917 

5.9% 

Identity  Pending 

255 

0.3% 

Total 

100,465 

100.0% 

Targeted  Disability 

Population 

Percent 

Disability 

553 

0.6% 

No  Disability 

99,912 

99.4% 

Total 

100,465 

100.0% 

Age 

Population 

Percent 

40  and  Over 

83,249 

82.9% 

Under  40 

17,216 

17.1% 

Total 

100,465 

100.0% 

Veteran  Status 

Population 

Percent 

Veteran 

34,200 

34.0% 

Non-Veteran 

66,265 

66.0% 

Total 

100,465 

100.0% 

Table  2-17  above  provides  a  demographic  breakout  of  the  100,465  NSPS  employees  that  were 
both  rated  and  payout  eligible  in  2007.  The  median  rating  for  each  subgroup  in  the  table  was  3. 
Mean  ratings  for  each  subgroup  across  pay  pools  are  not  considered  appropriate  performance 
measures,  as  explained  above,  and  are  not  reported.  Instead,  the  analysis  uses  linear  regression 
to  examine  differences  for  subgroups.  Regression  provides  the  opportunity  to  examine 
subgroup  effects  while  controlling  for  other  variables  that  affect  ratings.  The  approach  helps 
avoid  potentially  incorrect  inferences  developed  from  simple  two-way  comparisons. 

2.3.1.  Linear  Regression  Analysis 

The  specific  type  of  regression  used  for  the  analysis  is  known  as  "stepwise"  regression  because 
variables  are  added  one  at  a  time,  in  order  of  decreasing  significance.  If  an  included  variable 
drops  below  the  significance  threshold  (0.05)  at  any  time  during  the  process,  it  is  removed  from 
the  regression.  The  regression  terminates  when  none  of  the  excluded  variables  meets  or 
surpasses  the  significance  threshold  when  added  to  the  regression. 
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Independent  variables  for  the  regression  are  developed  from  subgroup  identities  through  the 
use  of  indicator  (0/1)  variables.  For  those  categories  of  subgroups  with  only  two  possible  values 
like  gender,  a  single  indicator  variable  is  defined  for  one  of  the  two  values.  For  gender  an 
indicator  variable  is  created  for  Female  (e.g.,  Female=l),  but  not  for  Male.  In  this  case  Male  is 
the  default  value  for  the  subgroup  breakout,  and  membership  is  implied  when  Female  equals 
zero.  Each  category  of  subgroup  has  a  default  subgroup  not  explicitly  identified  in  the 
regression.  For  categories  of  subgroups  with  more  than  two  possible  values,  such  as  race,  an 
indicator  variable  is  created  for  each  of  the  subgroups  except  the  largest  one,  which  is  the 
default  subgroup.  For  race,  White  is  the  default  value,  and  the  estimated  regression  coefficients 
for  each  of  the  race  indicator  variables  (i.e.,  Black,  Asian,  American  Indian/Alaskan  Native) 
represent  differences  relative  to  the  White  subgroup.  Army  was  selected  as  the  default 
component  because  the  resulting  regression  produced  coefficients  for  the  remaining 
components  that  all  have  the  same  sign,  facilitating  component  comparisons.  In  the  stepwise 
regression  procedure  described  above,  indicator  variables  representing  subgroups,  such  as  race 
or  component,  enter  the  regression  as  a  group  and  leave  as  a  group. 

Additional  variables  considered  in  the  regression  modeling  are  listed  below,  where  a  'C 
designates  a  continuous  variable  and  T  designates  an  indicator  variable: 

■  Education  level  - 1  (Bach.,  Masters,  PhD) 

■  Component  -  I  (Army,  Navy,  Air  Force,  and  4th  Estate) 

■  Years  of  service  -  C 

■  Career  Group  -  (Standard,  Scientific  &  Engineering,  Medical,  and  Investigative/Protective) 

■  Work  Category  (Professional,  Manager,  Technician)  - 1 

■  Pay  pool  funding  level  -  C 

■  Base  pay  -  C 

■  Average  score  of  pay  pool  -  C 

■  Relative  position  in  pay  band  -  C  (ranging  from  0  to  1,  where  .5  is  halfway  through  the 
band) 

Table  2-18  reports  results  for  the  regression  of  performance  rating  on  the  explanatory 
individual  and  pay  pool-level  variables  selected  by  the  stepwise  procedure.  The  overall  R2  is 
typically  used  as  an  indicator  of  the  fit  of  the  regression  model  to  the  underlying  data.  This 
stepwise  regression  produced  an  R2  of  just  15.8%,  meaning  over  84%  of  the  variance  in 
individual  performance  ratings  could  not  be  explained  by  the  variables  listed  above.  While  a  low 
R2  detracts  from  the  confidence  of  the  results,  the  estimated  coefficients  still  indicate  the 
underlying  relationships  between  ratings  and  the  explanatory  variables  and  reinforce 
conclusions  reached  in  earlier  sections  of  this  report. 

The  regression  method  also  calculates  the  percentage  of  the  R2  contributed  by  each 
explanatory  variable.  Table  2-18  shows  the  contributions  to  R2  and  estimated  coefficients  for  all 
variables  or  groups  of  variables  that  contribute  more  than  .1%  to  the  R2.  The  most  significant 
variable  by  far  was  average  rating  in  the  pay  pool,  which  contributed  9.4%  to  the  R2.  Thus, 
average  pay  pool  rating  contributed  60%  (9.4/15.8)  of  the  overall  R2.  This  result  is  not  surprising 
given  the  histogram  of  average  performance  ratings  shown  in  Figure  2-14.  This  variable  helps 
account  for  the  great  range  in  average  scores  across  pay  pools  and  supports  the  importance  of 
equity  analysis  at  the  pay  pool  level. 

The  second  most  significant  variable  is  base  salary,  which  contributes  2.7%  to  the  overall  R2. 

This  result  is  also  consistent  with  the  positive  correlation  between  rating  and  base  salary 
documented  in  Section  2.2.8.  Together,  the  two  most  significant  variables  account  for  over  75% 
of  the  R2. 
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The  third  most  significant  variable,  years  of  service,  has  a  negative  coefficient.  This  result  may 
appear  counter-intuitive  because  experience  in  earnings  models  generally  has  a  positive  effect. 
However,  this  regression  is  modeling  performance  ratings  and  controls  for  the  effects  of  other 
variables  in  the  model.  That  is,  if  two  individuals  have  the  same  salaries  and  work  category  but 
differing  years  of  service,  shorter  service  time  may  be  an  indication  of  superior  skills  or  previous 
superior  performance  that  has  resulted  in  faster  career  advancement.  This  reasoning  is  one 
possible  explanation  for  the  negative  coefficient  for  years  of  service. 

Both  managers  (+.154)  and  technicians  (+.109)  fared  better  than  the  default  work  category, 
professional.  Females  (+.126)  fared  better  than  males,  while  all  the  service  components 
(especially  4th  Estate)  showed  negative  coefficients  when  compared  to  the  default  component, 
Army.  Among  the  race  subgroups  coefficients  were  negative,  and  the  coefficient  for  Blacks  (- 
.134)  was  the  most  negative. 


Table  2.18  -  Regression  Results  for  Rating 


2.3.2.  Conclusions 

The  pay  pool  process,  designed  to  foster  equity  and  fairness  within  the  pay  pool,  complicates 
analysis  of  performance  ratings  across  pay  pools  due  to  differences  in  work  demands,  the  mix 
of  jobs  and  experience,  and  the  application  of  the  common  performance  indicators  and 
benchmarks  in  a  local  context.  For  a  system-wide  perspective,  this  analysis  used  linear 
regression  with  pay  pool-level  and  individual  demographic  variables  to  explain  differences  in 
performance  ratings,  but  only  explained  15.8%  of  the  variance. 

As  a  group,  the  race  indicator  variables  explained  only  0.6%  of  the  variance  in  performance 
ratings.  However,  the  regression  confirmed  results  obtained  earlier  in  the  report:  differences  in 
the  average  ratings  of  employees'  pay  pools  drive  the  explanation  of  individual  rating 
differences,  and  higher  paid  individuals  tend  to  receive  higher  ratings. 
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2.4.  Migration  to  and  from  NSPS 

The  analysis  tracked  the  migration  of  employees  into  and  out  of  NSPS  positions  using  end-of- 
month  DCPDS  extract  files.  The  status  of  employees  with  an  NSPS  rating  for  FY2006  was 
baselined  with  information  from  the  January  2007  extract  and  then  compared  to  information 
from  the  January  2008  extract  (about  a  year  after  receipt  of  rating).  Employees  with  an  NSPS 
rating  for  FY2007  were  tracked  using  information  from  the  January  2008  extract  with  follow-up 
information  from  the  July  2008  extract  (about  eight  months  after  receipt  of  rating).  Four 
categories  tracked  the  migration:  1)  Remained  in  the  same  NSPS  position,  2)  Moved  to  another 
NSPS  position,  3)  Moved  to  a  non-NSPS  position,  or  4)  Separated  from  DoD.  Employees  were 
also  grouped  by  performance  rating  (satisfactory  vs.  unsatisfactory)  to  indicate  potential 
differences  in  migration  patterns,  especially  DoD  separations.  In  addition,  the  analysis 
compares  migration  for  NSPS  participants  to  those  in  DoD  who  were  not  in  NSPS. 

The  top  of  Table  2-19  shows  the  2007  migration  pattern  of  those  in  Spiral  1.1  who  received  an 
NSPS  rating  for  2006.  The  bottom  shows  the  same  information  for  non-NSPS  employees. 
Comparing  the  NSPS  and  non-NSPS  cohorts  shows  the  difference  in  the  percent  receiving 
unsatisfactory  ratings:  2.9  percent  of  the  NSPS  cohort  received  low  ratings  of  1  or  2  compared 
with  only  .06  percent  for  the  non-NSPS  cohort.  Flowever,  the  loss  rate  for  employees  with  a  low 
rating  was  much  higher  for  the  non-NSPS  cohort:  40.7  percent  versus  16.8  percent  for  NSPS.  As 
of  January  2007,  23  percent  of  the  total  NSPS  Spiral  1.1  cohort  had  moved  with  14.5  percent 
moving  to  another  NSPS  position,  2  percent  moving  to  non-NSPS  positions,  and  6.5  percent 
leaving  DoD. 


Table  2.19  -  2007  Migration 


Low  rating 

3  or  above 
Total 


Employees 


297 

9,909 

10,206 


Non-NSPS 

Employees 


Remained 
in  the 
Same  Unit 


183 

7,671 

7,854 


Remained 
in  the 
Same  Unit 


NSPS  Cohort 

Moved  to  Moved  to 
other  NSPS  Non-NSPS 

Position  Position 


54 

1,429 

1,483 


10 

191 

201 


Non-NSPS  Cohort 


Moved  to  Moved  to 
NSPS  Non-NSPS 

Position  Position 


Separated 
from  DoD 


50 


618 

668 


Separated 
from  DoD 


Loss  Rate 


16.8% 

6.2% 

6.5% 


Loss  Rate 


Percent 
with  Low 
Rating 


2.9% 


Percent 
with  Low 
Rating 


Low  rating 

3  or  above 
Total 


150 

256,995 

257,145 


77 

188,483 

188,560 


8 

37,274 

37,282 


12,152 

12,156 


61 

19,086 

19,147 


40.7% 

7.4% 

7.4% 


0.06% 


Table  2-20  displays  similar  migration  data  for  the  much  larger  2007  NSPS  cohort  that  includes 
all  of  Spiral  1.  The  percent  of  NSPS  employees  with  a  low  rating  dropped  from  2.9  percent  in 
2006  to  1.7  percent  in  2007.  The  NSPS  and  non-NSPS  loss  rates  for  employees  with  low  ratings 
were  also  similar:  13.2  percent  for  NSPS  and  12.6  percent  for  non-NSPS.  Both  NSPS  and  non- 
NSPS  employees  had  the  same  overall  loss  rates  of  3.4  percent,  which  represent  declines  of  3.1 
and  4.0  percentage  points,  respectively.  In  summary,  as  of  July  2008,  8.6%  of  the  NSPS  Spiral  1 
workforce  had  moved,  with  3.7  percent  moving  to  other  NSPS  positions,  1.6  percent  moving  to 
non-NSPS  positions,  and  3.4  percent  separating  from  DoD. 
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Table  2.20  -  2008  Migration 


NSPS  Cohort 


NSPS 

Employees 

Remained  in 
the  Same  Unit 

Moved  to  other 
NSPS  Position 

Moved  to  Non- 
NSPS  Position 

Separated 
from  DoD 

Loss  Rate 

Percent  with 
Low  Rating 

Low  rating 

1,644 

1,342 

29 

56 

217 

13.2% 

1.7% 

3  or  above 

96,738 

88,568 

3,565 

1,498 

3,107 

3.2% 

Total 

98,382 

89,910 

3,594 

1,554 

3,324 

3.4% 

Non-NSPS  Cohort 

Non-NSPS 

Remained  in 

Moved  to  NSPS 

Moved  to  Non- 

Separated 

Loss  Rate 

Percent  with 

Employees 

the  Same  Unit 

Position 

NSPS  Position 

from  DoD 

Low  Rating 

Low  rating 

562 

421 

64 

6 

71 

12.6% 

0.24% 

3  or  above 

237,289 

190,720 

31,228 

7,220 

8,121 

3.4% 

Total 

237,851 

191,141 

31,292 

7,226 

8,192 

3.4% 

2.5.  Rating  Reconsideration  and  EEO  Complaint  Results 

2.5.1.  Rating  Reconsiderations 

Of  the  100,465  NSPS  employees  who  received  a  performance  rating  for  FY2007,  2.3  percent 
requested  a  reconsideration  of  rating.  Table  2-21  shows  the  results  for  each  component. 
Overall,  a  third  of  the  reconsideration  requests  were  granted.  The  Fourth  Estate  had  the  highest 
reconsideration  approval  rate  (45.5  percent)  while  the  Air  Force  had  the  lowest  (28.0  percent) 


Table  2.21  -  Rating  Reconsiderations/  January  2008  Results 


Component 

Total  Requested 

Granted 

Percent  Granted 

Army 

810 

282 

34.8% 

DON 

319 

124 

38.9% 

Air  Force 

973 

272 

28.0% 

Fourth  Estate 

200 

91 

45.5% 

Total 

2,302 

769 

33.4% 

2.5.2.  EEO  Complaints 

Figure  2-15  shows  the  number  of  DoD  complaints  of  discrimination  received  by  the 
Investigations  and  Resolution  Division  (IRD)  of  the  Civilian  Personnel  Management  Service  for 
the  last  three  fiscal  years.  The  NSPS/Non-NSPS  breakout  reflects  the  classification  of  the 
originating  organization  when  the  complaint  was  filed.  Thus,  some  of  the  complaints  shown  as 
Non-NSPS  in  FY2007  and  FY2008  are  from  employees  who  have  since  converted  to  NSPS 
positions.  In  FY2007,  IRD  received  22  NSPS  complaints,  less  than  1  percent  of  the  total,  and  in 
FY2008,  154  complaints,  5.5  percent  of  the  total. 


Total  EEO  Complaints 


4,000 

3.500 
3,000 

2.500 
2,000 

1.500 
1,000 

500 

0 


2 22 

3,482 

3,323 

154 

2,637 

FY2006 


FY2007 


FY2008 


■  NSPS 
□  Non-NSPS 


Figure  2.15  -  DoD  EEO  Complaints 
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Figure  2.16  compares  the  composition  of  NSPS  and  Non-NSPS  complaints  by  basis.  Despite  the 
smaller  NSPS  sample  size,  the  NSPS  and  Non-NSPS  distributions  by  basis  are  similar.  From  this 
perspective,  there  is  no  indication  that  NSPS  is  causing  the  mix  of  complaints  by  the  eight  basis 
categories  to  be  very  different  from  the  overall  mix. 


2008  Cases  by  Basis 


□  Non-NSPS 
■  NSPS 


Figure  2,16  -  NSPS  and  Non-NSPS  Complaints  by  Basis 


The  next  section  discusses  the  analysis  of  employee  performance  plans. 


2.6.  Analysis  of  Employee  Performance  Plans 

An  important  element  of  the  National  Security  Personnel  System  is  the  employee  performance 
plan.  Supervisors  use  performance  plans  to: 

■  Document  the  objectives  that  guide  employee  performance  throughout  the  performance 
period 

■  Provide  ratings  of  employee  performance  relative  to  these  objectives 

■  Provide  justifications  of  these  ratings 

Employees  have  the  opportunity  to  contribute  to  the  performance  plans  by  providing  a  self- 
assessment  in  which  they  explain  how  they  perceive  they  performed  relative  to  their 
objectives.  The  information  provided  by  supervisors  and  employees  on  the  performance  plans 
is  then  used  in  the  pay  pool  panel  process  to  determine  final  performance  ratings  and  share 
allocations. 

Given  the  significance  of  the  information  provided  in  the  performance  plans,  the  NSPS  PEO 
requested  an  evaluation  of  the  performance  plans  developed  under  NSPS  to  determine  the 
extent  to  which  the  performance  plans  are  effectively  prepared  and  meet  the  needs  of  NSPS. 
The  specific  research  questions  that  guided  this  evaluation  were: 

■  To  what  extent  did  the  objectives  adhere  to  the  "SMART"  framework  (i.e.,  specific, 
measurable,  aligned,  realistic,  and  timed)? 

■  To  what  extent  did  the  objectives  reflect  appropriate  difficulty  levels  and  meaningful 
outcomes  for  the  band  level  and  job  series  of  the  employee? 

■  To  what  extent  were  the  objectives  similar  within  organizations  and  components  for  the 
same  job  series? 

■  To  what  extent  did  the  objectives  increase  in  difficulty  for  higher  bands? 

■  To  what  extent  was  the  information  provided  in  the  performance  plans  adequate? 
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This  section  describes  the  methodology  used  to  evaluate  the  performance  plans,  the  results  of 
the  evaluation,  and  conclusions  of  the  study. 

2.6.1.  Evaluation  Methodology 

This  section  describes  the  procedures  for  choosing  the  sample  of  performance  plans  used  in 
this  evaluation  and  the  procedures  used  to  evaluate  these  performance  plans. 

Sample 

The  study  evaluates  240  performance  plans.  The  requirements  for  the  composition  of  the 
sample  of  performance  plans  included  a  range  of  job  series,  pay  schedules,  pay  bands,  and 
organizations  within  each  of  the  four  DoD  components.  The  specific  job  series  requested  are 
ones  with  which  the  analysis  team  has  past  experience  so  that  they  could  knowledgeably 
evaluate  job-specific  information  on  the  performance  plans.  NSPS  performance  plans  were 
chosen  to  meet  the  sample  requirements.  The  characteristics  of  the  final  sample  of 
performance  plans  are  presented  in  Table  2.22. 


Table  2.22  -  Performance  Plan  Sample  Characteristics 


Characteristics 

n  (percent) 

Component 

Army 

64  (27%) 

Navy 

60  (25%) 

Air  Force 

76  (32%) 

Fourth  Estate 

40  (17%) 

Job  Series 

201:  Human  Resources  Management 

24  (10%) 

301:  Miscellaneous  Administration  &  Program 

1  (0.4%) 

303:  General  Support 

15  (6%) 

304:  Information  Receptionist 

1  (0.4%) 

318:  Secretary 

1  (0.4%) 

343:  Management  and  Program  Analyst 

23  (10%) 

511:  Auditing 

24  (10%) 

560:  Budget  Analysis 

24  (10%) 

802:  Engineering  Technician 

24  (10%) 

810:  Civil  Engineering 

24  (10%) 

830:  Mechanical  Engineering 

24  (10%) 

1102:  Contracting 

32  (13%) 

1702:  Education  and  Training  Technician 

23  (10%) 

Pay  Schedule  and  Band 

YA1 

31  (13%) 

YA2 

50  (21%) 

YA3 

8  (3%) 

YB1 

20  (8%) 

YB2 

20  (8%) 

YC2 

22  (9%) 

YC3 

17  (7%) 

YD1 

8  (3%) 

YD2 

14  (6%) 

YD3 

14  (6%) 

YE2 

5  (2%) 

YE3 

11  (5%) 

YE4 

8  (3%) 

YF2 

6  (3%) 

YF3 

6  (3%) 
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Evaluation  Procedure 

The  performance  plan  evaluation  team  developed  a  set  of  evaluation  criteria  to  serve  as  a 
foundation  for  the  evaluation  and  to  address  the  study's  primary  research  questions.  Appendix 
B  contains  the  criteria  used  to  evaluate  the  performance  plans. 

After  the  evaluation  criteria  were  developed,  a  team  of  industrial/organizational  psychologists 
was  trained  as  assessors  to  use  the  criteria  to  evaluate  each  performance  plan.  After  each 
assessor  independently  evaluated  the  same  subset  of  performance  plans  using  the  evaluation 
criteria,  they  compared  individual  ratings,  discussing  areas  of  discrepancy  to  derive  a  final  rating 
for  each  performance  plan.  They  repeated  this  process  with  another  subset  of  performance 
plans  to  ensure  that  each  assessor  interpreted  and  applied  the  criteria  in  a  similar  manner. 
Once  a  high  level  of  agreement  on  ratings  was  attained,  the  remaining  performance  plans  were 
divided  so  that  the  same  assessor  evaluated  all  performance  plans  within  a  job  series.  This 
approach  was  necessary  because  some  of  the  evaluation  criteria  required  the  analysis  to 
compare  objectives  across  bands  within  a  job  series. 

The  lead  assessor  evaluated  a  random  set  of  each  of  the  remaining  performance  plans  for 
quality  control  to  ensure  that  the  team  assessors  remained  calibrated  throughout  the  task  and 
did  not  inadvertently  change  their  interpretations  of  the  evaluation  criteria.  The  lead  assessor 
periodically  discussed  any  issues  that  arose  with  the  team,  jointly  deciding  process  solutions. 
These  steps  ensured  that  the  assessor  team  applied  the  criteria  consistently  throughout  the 
evaluation  process. 

2.6.2.  Results 

The  240  performance  plans  yielded  a  total  of  893  objectives,  resulting  in  a  mean  of  3.7 
objectives  per  performance  plan.  This  section  presents  the  results  of  the  evaluation  according 
to  the  topics  of  the  primary  research  questions. 

SMART  Objectives 

Each  objective  was  evaluated  on  the  extent  to  which  it  met  each  element  of  the  SMART 
framework  (i.e.,  specific,  measurable,  aligned,  realistic,  and  timed).  These  elements  are  further 
explained  below. 

■  Specific:  Objectives  should  describe  specific  outcomes  or  results  to  be  achieved. 

■  Measurable:  Objectives  should  be  measurable  and  include  a  method  or  procedure  to 
observe,  verify,  assess,  or  record  the  result. 

■  Aligned:  Objectives  should  be  aligned  with  higher-level  organizational  goals  to  ensure 
that  employee  performance  contributes  to  the  achievement  of  organizational  success. 

■  Realistic:  Objectives  should  be  challenging,  but  they  also  should  be  realistic  and 
attainable  by  the  employee.  Employees  must  have  control  over  any  objective  that  is 
created. 

■  Timed:  Objectives  should  include  a  timeframe  for  completion,  such  as  deadlines,  cycles, 
or  schedules. 

The  results  of  the  evaluation  of  the  objectives  against  the  SMART  framework  are  presented  in 
Table  2.23. 

These  results  indicate  that: 

■  Fifty-eight  percent  of  the  objectives  were  highly  specific,  including  very  specific  and 
observable  actions,  behaviors,  or  achievements  that  clearly  identified  key  results  and 
outcomes.  In  contrast,  27%  included  moderately  specific  and  observable  actions, 
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behaviors,  or  achievements  that  were  somewhat  vague  or  broad,  and  14%  were  written 
in  generic  terms  with  no  specific  or  observable  actions,  behaviors,  or  achievements. 


Table  2.23  -  Elements  of  SMART  Framework 


SMART 

Frequency  of  rating 

Element 

n  (percent) 

Specific 

Not  Available 

Not  Specific 

Moderately  Specific 

Highly  Specific 

2  (0.2%) 

130  (14%) 

242  (27%) 

519  (58%) 

Measurable 

Not  Available 

Not  Measurable 

Moderately 

Measurable 

Highly  Measurable 

4  (0.5%) 

223  (25%) 

259  (29%) 

407  (46%) 

Aligned 

Not  Available 

Not  Aligned 

Moderately  Aligned 

Strongly  Aligned 

5  (0.6%) 

5  (0.6%) 

33  (4%) 

850  (95%) 

Realistic 

Not  Available 

Not  Realistic 

Somewhat  Realistic 

Very  Realistic 

5  (0.6%) 

1  (0.1%) 

6  (0.7%) 

881  (99%) 

Timed 

Not  Available 

Did  Not  Include 
Timeframe 

Included  Timeframe 

— 

2  (0.2%) 

425  (48%) 

466  (52%) 

— 

n  =  893  objectives. 

■  Less  than  half  (46%)  of  the  objectives  were  highly  measurable,  including  an  exact  and 
precise  method  or  procedure  for  assessing  or  verifying  the  quality  of  the  objective. 
Twenty-nine  percent  were  moderately  measurable,  including  an  approximate,  vague,  or 
imprecise  method  or  procedure  for  assessing  or  verifying  the  quality  of  the  objective.  One 
quarter  (25%)  of  the  objectives  did  not  include  a  method  or  procedure  for  assessing  or 
verifying  the  quality  of  the  objective. 

■  The  majority  (95%)  of  the  objectives  were  strongly  aligned  to  higher  level  goals, 
demonstrating  a  very  clear,  direct,  and  strong  linkage  to  the  organizational  mission  or 
relevant  strategic  goals. 

■  The  majority  (99%)  of  objectives  were  very  realistic  and  attainable. 

■  Approximately  half  (52%)  of  the  objectives  included  a  timeframe  for  completion  and 
approximately  half  (48%)  did  not  include  a  timeframe. 


Appropriateness  of  Objectives  for  the  Band  Level  and  Job  Series 

Adhering  to  the  SMART  framework  is  necessary,  but  not  sufficient,  for  effective  objectives. 
Objectives  should  be  written  at  the  appropriate  difficulty  for  the  band  level  of  the  employee 
and  should  reflect  meaningful  outcomes  for  the  employee's  job  series.  Objectives  should  not 
focus  on  relatively  trivial  requirements  of  the  job.  The  results  of  the  evaluation  of  the  difficulty 
level  and  meaningfulness  of  the  objectives  are  presented  in  Table  2.24. 


Table  2.24  -  Appropriateness  of  Objectives 


Difficulty  Level 
Meaningfulness 


Not  Available 
5  (0.6%) 


Not  Available 
5  (0.6%) 


Frequency  of  Rating 
n  (percent) 


Too  Low 
51  (6%) 


Not  Meaningful 
Outcome 
12  (1%) 


Appropriate 
836  (94%) 


Moderately  Meaningful 
Outcome 
187  (21%) 


Too  High 
1  (0.1%) 


Very  Meaningful 
Outcome 
689  (77%) 


n  =  893  objectives. 


These  results  indicate  that: 

■  The  majority  (94%)  of  objectives  were  written  at  an  appropriate  level  of  difficulty  for  the 
band  level. 
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■  Approximately  77%  of  the  objectives  reflected  a  very  meaningful  outcome  that  was 
relevant,  key,  and  significant  to  the  job  series.  Less  than  a  quarter  (21%)  of  the  objectives 
were  moderately  meaningful,  and  only  1%  of  the  objectives  reflected  outcomes  that  were 
irrelevant  or  insignificant  to  the  tasks  of  the  job. 

Similarity  of  Objectives 

In  addition  to  evaluating  the  performance  plans  on  an  individual  level,  the  analysis  evaluated 
sets  of  objectives  for  employees  in  the  same  job  series  and  band  level  on  the  similarity  of  the 
objectives.  This  approach  was  important  to  identify  the  extent  to  which  employees  in  similar 
roles  were  being  evaluated  on  similar  objectives. 

The  sets  of  performance  plans  spanned  a  range  of  job  series,  band  levels,  pay  schedules, 
organizations,  and  components.  The  number  of  performance  plans  within  each  set  ranged  from 
3  to  15  plans.  The  results  of  this  evaluation  are  presented  in  Table  2.25. 


Table  2.25  -  Similarity  of  Objectives 


Frequency  of  Rating 
n  (percent) 

Within 

Not  Similar 

Somewhat  Similar 

Very  Similar 

Organizations  1 

0  (0%) 

9  (43%) 

12  (57%) 

Within 

Not  Similar 

Somewhat  Similar 

Very  Similar 

Components 2 

3  (9%) 

11  (32%) 

20  (59%) 

1  n  =  21  sets  of  performance  plans. 

2  n  =  34  sets  of  performance  plans 

These  results  indicate  that: 

■  For  over  half  of  the  within-organization  and  within-component  sets  (57%  and  59%, 
respectively),  the  performance  plans  were  very  similar  to  each  other,  with  over  75%  of 
the  performance  plans  within  these  sets  containing  objectives  that  reflected  similar 
outcomes  or  key  results. 

■  For  43%  of  the  within-organization  sets  and  approximately  one  third  (32%)  of  the  within- 
component  sets,  the  performance  plans  were  somewhat  similar  to  each  other,  with  40% 
to  75%  of  the  performance  plans  within  these  sets  containing  objectives  that  reflected 
similar  outcomes  or  key  results. 

■  For  9%  of  the  within-component  sets,  the  performance  plans  were  not  similar  to  each 
other,  with  less  than  40%  of  the  performance  plans  within  these  sets  containing 
objectives  that  reflected  similar  outcomes  or  key  results.  None  of  the  within-organization 
sets  were  evaluated  as  "not  similar." 

Progression  of  Objective  Difficulty 

The  analysis  evaluated  sets  of  objectives  for  employees  in  the  same  job  series  on  the  extent  to 
which  the  objectives  progressed  in  difficulty  from  lower  bands  to  higher  bands.  This  process 
was  important  to  determine  whether  the  difficulty  of  an  objective  for  any  given  employee 
within  the  same  job  series  was  at  the  appropriate  level  relative  to  others  in  different  bands 
throughout  the  same  job  series. 

As  for  evaluation  of  the  similarity  of  objectives  discussed  above,  the  progression  of  difficulty 
level  was  assessed  within  organizations  and  within  components.  The  sets  of  performance  plans 
spanned  a  range  of  job  series,  band  levels,  pay  schedules,  organizations,  and  components.  The 
number  of  performance  plans  within  each  set  ranged  from  3  to  15  plans.  The  results  of  this 
evaluation  are  presented  in  Table  2.26. 
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Table  2.26  -  Progression  of  Objective  Difficulty 


Frequency  of  Rating 
n  (percent) 


Within 

Organization  1 2 

Did  Not  Increase  in  Difficulty 

0  (0%) 

Increased  in  Difficulty  to  a 
Moderate  Extent 

2  (9%) 

Increased  in  Difficulty  to  a 
Great  Extent 

21  (91%) 

Within 
_  .  2 

Did  Not  Increase  in  Difficulty 

Increased  in  Difficulty  to  a 
Moderate  Extent 

Increased  in  Difficulty  to  a 
Great  Extent 

Component 

0  (0%) 

1  (8%) 

12  (92%) 

1  n  =  23  sets  of  performance  plans. 


2  n  =  13  sets  of  performance  plans 

These  results  indicate  that: 

■  For  the  majority  of  the  within-organization  and  within-component  sets  (91%  and  92%, 
respectively),  the  performance  plans  increased  in  difficulty  to  a  great  extent,  with  at  least 
75%  of  the  performance  plans  at  each  band  level  being  more  difficult  than  the  majority  of 
the  performance  plans  at  the  lower  band  level. 

■  For  less  than  10%  of  the  within-organization  and  within-component  sets  (9%  and  8%, 
respectively),  the  performance  plans  increased  in  difficulty  to  a  moderate  extent,  with  at 
least  50%  of  the  performance  plans  at  each  band  level  being  more  difficult  than  the 
majority  of  the  performance  plans  at  the  lower  band  level. 

■  None  of  the  sets  failed  to  progress  in  difficulty  to  at  least  a  moderate  extent  from  lower 
to  higher  bands. 


Adequacy  of  Performance  Descriptions 

The  information  that  employees  and  supervisors  provide  on  the  performance  plans  is  the 
foundation  for  important  decisions  on  performance  ratings  and  share  allocations.  As  such,  it 
was  important  to  assess  the  adequacy  of  the  information  provided  to  ensure  that  the 
information  meets  the  needs  of  supervisors  and  pay  pool  panels  in  making  these  decisions. 

The  analysis  evaluated  two  sections  of  the  performance  plans  to  assess  the  adequacy  of  the 
performance  descriptions:  the  self-assessments  written  by  the  employees  to  describe  their 
performance,  and  the  narratives  provided  by  the  supervisors  to  justify  the  ratings  they 
provided.  Self-assessments  were  evaluated  on  the  extent  to  which  employees  provided 
adequate  descriptions  of  the  results  they  achieved  and  how  they  accomplished  each  objective. 
The  supervisor  narratives  were  evaluated  on  the  extent  to  which  they  provided  adequate 
justifications  for  the  ratings  they  provided  and  for  any  adjustments  made  to  account  for 
contributing  factors.  These  results  are  presented  in  Table  2.27. 


Self  Assessments  1 

Rating 
Narrative  1 
Contributing 
Factor  Adjustment 


Table  2.27  -  Adequacy  of  Information  in  Performance  Plans 


Frequency  of  Rating 
n  (%) 


1  n  =  240  performance  plans. 

2  n  =  153  performance  plans. 


Not  Available 

Inadequate 

Adequate 

Very  Extensive 

9  (4%) 

15  (6%) 

56  (23%) 

160  (67%) 

Not  Available 

Inadequate 

Adequate 

Very  Extensive 

1  (0.4%) 

42  (18%) 

59  (25%) 

138  (58%) 

— 

Inadequate 

Adequate 

Very  Extensive 

— 

40  (26%) 

71  (46%) 

42  (27%) 

These  results  indicate  that: 

■  Approximately  two-thirds  (67%)  of  employees  provided  very  extensive  descriptions  of 
their  performance  relative  to  the  performance  objectives.  Twenty-three  percent  provided 
adequate  descriptions  and  only  6%  provided  inadequate  descriptions  of  their 
performance. 


May  15,  2009 


2-31 


2.  Analysis  of  DCPDS/CWB  Data 


NSPS  Spiral  1  Evaluation  Report 


m  Over  half  (58%)  of  the  supervisors  provided  very  extensive  justifications  for  the 
performance  ratings  they  provided  to  employees.  One  quarter  (25%)  of  the  supervisors 
provided  adequate  justifications  and  18%  of  the  supervisors  provided  inadequate 
justifications  of  the  performance  ratings. 

■  Approximately  64%  of  the  performance  plans  used  contributing  factors  and  36%  did  not. 
Of  the  plans  that  used  contributing  factors,  71%  identified  contributing  factors  for  more 
than  half  of  the  objectives  and  29%  identified  contributing  factors  for  half  or  fewer  of  the 
objectives.  Approximately  one  quarter  (27%)  of  the  supervisors  provided  very  extensive 
justifications  to  support  their  decisions  regarding  whether  to  adjust  the  performance 
ratings  due  to  the  contributing  factors.  Slightly  less  than  half  (46%)  provided  adequate 
justifications  and  approximately  one  quarter  (26%)  provided  inadequate  justifications. 

2.6.3.  Conclusions 

In  summary,  the  purpose  of  this  study  was  to  conduct  an  evaluation  of  employee  performance 
plans  developed  under  NSPS.  As  such,  the  analysis  evaluated  240  performance  plans 
representing  a  range  of  job  series,  pay  schedules,  band  levels,  and  organizations  within  the  four 
components  to  address  the  following  questions: 

■  To  what  extent  did  the  objectives  adhere  to  the  "SMART"  framework  (i.e.,  specific, 
measurable,  aligned,  realistic,  and  timed)?  Over  90%  of  the  objectives  were  strongly 
aligned  and  very  realistic.  However,  much  fewer  were  highly  specific,  highly  measurable, 
and  included  a  timeframe  for  completion.  This  represents  an  opportunity  for 
improvement  within  NSPS  organizations. 

■  To  what  extent  did  the  objectives  reflect  appropriate  difficulty  levels  and  meaningful 
outcomes  for  the  employee?  Most  of  the  objectives  were  written  at  an  appropriate  level 
of  difficulty  for  the  band  level  and  reflected  a  very  meaningful  outcome  that  was  relevant, 
key,  and  significant  to  the  job  series. 

■  To  what  extent  were  the  objectives  similar  within  organizations  and  components  for  the 
same  job  series?  This  question  was  addressed  by  evaluating  sets  of  performance  plans, 
with  each  set  including  performance  plans  for  employees  in  the  same  job  series  and  band 
level.  Over  half  of  the  within-organization  and  within-component  sets  of  performance 
plans  were  very  similar  to  each  other.  Very  few  of  the  within-component  sets  of 
performance  plans  were  evaluated  as  "not  similar"  and  none  of  the  within-organization 
sets  of  performance  plans  were  evaluated  as  "not  similar." 

■  To  what  extent  did  the  objectives  progress  in  difficulty  for  higher  bands?  This  question 
was  addressed  by  evaluating  sets  of  performance  plans,  with  each  set  including 
performance  plans  for  employees  in  the  same  job  series  but  in  different  band  levels.  Over 
90%  of  objectives  in  the  within-organization  and  within-component  sets  increased  to  a 
great  extent  in  difficulty  from  lower  bands  to  higher  bands.  None  of  the  sets  failed  to 
progress  in  difficulty,  at  least  to  a  moderate  extent. 

■  To  what  extent  was  the  information  provided  in  the  performance  plans  adequate?  The 
majority  of  employees  provided  very  extensive  descriptions  of  their  performance  relative 
to  the  performance  objectives.  Over  half  of  the  supervisors  provided  very  extensive 
justifications  for  the  overall  performance  ratings  they  provided  to  employees.  In  contrast, 
only  about  one  quarter  of  the  supervisors  provided  very  extensive  justifications  to 
support  their  decisions  regarding  whether  to  adjust  the  performance  ratings  due  to  the 
contributing  factors.  This  represents  another  opportunity  for  improvement  within  NSPS. 
Supervisors  need  to  describe  how  the  employee's  performance  related  to  the 
contributing  factor(s)  associated  with  a  given  objective. 
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3.1.  Introduction 

As  part  of  the  evaluation,  members  of  the  study  team  conducted  12  site  visits,  consisting  of  58 
focus  groups  with  635  employees  and  supervisors  and  53  interviews  with  Commanders  and 
Directors,  Performance  Review  Authorities,  members  of  pay  pool  panels  (in  both  panel  and 
senior  manager  roles),  and  implementation  teams.  The  purpose  of  the  site  visits  was  to  collect 
information  regarding  perspectives  on  whether  NSPS  is  on  track  to  meet  its  goals,  and  to 
identify  aspects  of  NSPS  and  its  operation  that  warrant  attention  and  possible  adjustments.  This 
chapter  provides  a  detailed  discussion  of  the  major  themes  identified  from  the  interviews  and 
focus  groups,  as  well  as  a  review  of  the  documentation  collected  from  each  site  regarding  NSPS 
implementation  and  operation.  The  study  team  identified  16  themes  that  reflect  the  broadest 
areas  of  input  from  across  all  site  visit  groups.  To  arrive  at  these  themes,  the  study  team  first 
reviewed  the  transcripts  from  all  111  focus  groups  and  interviews,  thereby  identifying 
approximately  4,000  points,  or  themes,  made  by  the  participants.  Next,  the  study  team 
eliminated  redundancies,  reducing  the  set  to  211  themes.  Finally,  the  study  team  conducted 
additional  analyses  to  identify  the  strongest,  most  consistent  findings  across  all  focus  groups 
and  interviews.  These  16  themes  are  the  major  findings  from  the  site  visits.  Appendix  C 
provides  additional  detail  on  the  methodology  used  to  conduct  these  analyses. 

The  16  themes  are  presented  by  KPP,  with  the  following  structure: 

■  An  overview  of  the  theme,  including  detailed  perspectives 

■  Quotations  from  the  focus  group  and  interview  transcripts  with  additional  detail  for  the 
sub-components  of  each  major  theme 

3.2.  Information  on  the  Sites 

Table  3.1  indicates  locations  for  the  12  site  visits  and  the  number  of  focus  groups  and 
interviews  by  location  and  cohort. 


Table  3.1-  Number  of  Focus  Groups  and  Interviews  by  Location  and  Cohort 


Cohort 

Location 

Employee 

Supervisor 

Pay  Pool 
Panel 

Imp.  Team 

Commander/ 

Director 

PRA 

Total 

CHRA  North  Central  CPOC 

2 

1 

1 

2 

1 

0 

7 

Tripler  Army  Medical  Center 

3 

2 

1 

1 

0 

1 

8 

Army  Engineering  District, 
Jacksonville 

3 

2 

2 

1 

1 

1 

10 

HQUSMC 

3 

2 

2 

1 

1 

1 

10 

PACFLT 

2 

1 

3 

1 

1 

8 

SPAWAR  SYSCEN 

5 

2 

2 

1 

1 

1 

12 

Scott  AFB 

3 

2 

3 

1 

0 

1 

10 

Offutt  AFB 

3 

2 

2 

2 

1 

0 

10 

Wright-Patterson  AFB 

3 

3 

2 

1 

0 

1 

10 

Defense  Security  Cooperation 
Agency 

3 

1 

1 

1 

2 

2 

10 

Missile  Defense  Agency 

3 

1 

2 

1 

0 

0 

7 

DoD  IG 

4 

2 

2 

1 

0 

0 

9 

Total 

37 

21 

23 

14 

8 

8 

111 
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3.3.  Abstract  of  Findings 

The  16  themes  are  identified  below  with  a  summary  of  findings  for  each. 

3.3.1.  Themes  Related  to  KPP 1:  High-Performing  Workforce  and  Management 

Theme  1.  Link  Between  Performance  and  Pay  Is  Moderated  by  Several  Factors 

There  were  varied  assessments  about  the  link  between  performance  and  pay  under  NSPS. 
Themes  include  perceptions  of  an  emphasis  on  3  ratings,  the  inclusion  of  some  marginal 
performers  and  solid  performers  in  the  same  rating  level,  the  importance  of  job  visibility  and 
supervisor  writing  skills  to  high  ratings,  and  the  effects  of  share  allocations  and  salary/bonus 
splits  on  payouts. 

Theme  2.  Employees'  Negative  Perception  of  a  3  Rating 

NSPS'  five-point  rating  scale  with  "3-valued  performer"  as  a  solid  level  of  performance  is 
difficult  for  many  in  the  workforce  to  accept  and  has  led  to  reports  of  lower  morale.  Further, 
some  employees  are  concerned  about  the  impact  of  a  rating  of  3  on  their  career  opportunities. 
A  majority  stated  they  cannot  articulate  what  they  must  do  to  exceed  a  3  rating. 

Theme  3.  Effects  on  Recruitment  and  Retention 

Employees  view  the  ability  to  move  more  quickly  through  a  pay  band  as  a  benefit  of  NSPS. 
Other  advantages  include  the  ability  to  negotiate  salaries  for  new  hires  and  more  flexibility  in 
recruiting  and  retaining  employees  with  higher-level  expertise.  Challenges  to  recruitment  and 
retention  include  the  five  percent  limit  on  salary  increases  for  reassignments,  pay  band  caps, 
perceived  pay  inequities  between  current  employees  and  new  hires,  competing  GS 
opportunities,  a  perception  of  receiving  smaller  increases  than  comparable  GS  employees,  a 
lack  of  understanding  of  how  to  advance  in  a  pay-banded  system,  declining  morale  and  job 
satisfaction,  and  a  perception  that  supervisors'  inability  to  justify  ratings  leads  to  lower  payouts. 
While  the  hiring  process  is  reported  to  be  difficult  and  time-consuming,  it  is  not  reported  to  be 
more  difficult  than  under  previous  systems.  However,  NSPS's  broad  pay  bands  have  made  it 
more  difficult  to  hire  for  specific  pay  ranges  and  specific  specialty  positions  (e.g.,  IT 
professionals).  Another  pay-related  challenge  includes  organizational  tailoring  within  broad  pay 
bands  (e.g.,  use  of  "pay  lanes")  for  NSPS  vacancy  announcement  salaries.  Senior  leaders  and 
supervisors  generally  believe  that  neither  the  quality  of  applicants  nor  employee  performance 
has  changed;  it  may  be  too  early  to  see  any  effects. 

Theme  4.  Confusion  about  Advancement  within  and  between  Pay  Bands 

Many  in  the  workforce  are  uncertain  about  how  to  advance  within  a  pay-banded  system.  While 
some  employees  appreciate  wide  pay  bands  for  the  opportunities  they  offer,  and  some 
supervisors  expressed  appreciation  for  the  increased  flexibility  in  making  assignments,  still 
other  employees  lack  information  on  how  to  progress.  Many  employees  seeking  career 
advancement  are  confused  about  how  to  reconcile  GS  job  descriptions  with  NSPS'  broad  pay 
bands. 

Theme  5.  Payout  Amounts  Were  Generally  Viewed  Favorably,  but  the  Government-Wide  Pay 

Increase.  Effects  on  Retirement,  and  GS-Equivalence  Are  Issues 

The  workforce  generally  appreciated  the  value  of  their  payouts  despite  some  dissatisfaction 
with  performance  ratings.  However,  employees  at  all  levels  expressed  concerns  about  having  a 
portion  of  the  government-wide  pay  increase  at  risk,  the  effects  of  bonuses  on  retirement 
"High  Three"  calculations,  and  the  value  of  payouts  relative  to  GS. 
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Theme  6.  Mixed  Value  of  Interim  Reviews  and  Self-Assessments 

There  were  varied  views  on  the  performance  evaluation  process.  While  some  supervisors  and 
employees  appreciated  that  NSPS  has  resulted  in  more  regular  and  formal  performance 
feedback,  others  indicated  that  the  interim  reviews  did  not  always  provide  specific  suggestions 
for  improvement  and  were  not  always  an  indication  of  the  year-end  performance  rating.  Some 
employees  viewed  self-assessments  as  an  opportunity  to  showcase  contributions  to  their 
organization's  mission,  while  others  viewed  them  primarily  as  a  challenging  writing  assignment. 

Theme  7.  Five  Percent  Cap  on  Reassignment  Pay  Increases  Impacts  Incentive  to  Assume 

Supervisory  Positions 

The  five  percent  pay  increase  limit  for  reassignments  is  said  to  discourage  eligible  employees 
from  assuming  supervisory  positions  with  their  perceived  higher  workloads  under  NSPS.  Eligible 
employees  indicate  reluctance  to  assume  a  supervisory  position  for  a  pay  raise  that  is  not 
perceived  to  be  consistent  with  the  additional  NSPS  workload.  Supervisors  cited  similar 
concerns  and  noted  examples  of  supervisors  who  left  or  are  considering  leaving  their 
organizations  because  of  the  workload. 

3.3.2.  Themes  Related  to  KPP  3:  Credible  and  Trusted  System 

Theme  8.  Pay  Pool  Panel  Processes  Lack  Transparency 

While  many  organizations  had  good  communication  with  the  workforce,  uneven  understanding 
of  pay  pool  panel  procedures  and  incomplete  information  on  organization-wide  ratings  and 
payouts  hindered  acceptance  of  NSPS  as  a  credible  and  trusted  system. 

Theme  9.  Differences  Across  Pay  Pools  Affect  Perceptions  of  Fairness 

An  issue  related  to  pay  pool  panel  transparency  is  perceived  differences  in  processes  and 
results  across  pay  pools,  raising  concerns  about  unfair  advantage  and  reward.  Performance 
Review  Authorities  are  concerned  with  ensuring  the  pay  pool  panel  system  is  fair,  transparent, 
and  equitable,  and  they  were  generally  satisfied  with  the  processes  and  outcomes. 

Theme  10.  Employees  and  Senior  Managers  Differ  in  their  Interpretations  of  the  Number  of 

Reconsiderations 

There  is  mixed  understanding  and  utilization  of  the  reconsideration  process.  Many  employees 
took  a  "wait-and-see"  attitude  about  NSPS  before  pursuing  reconsideration.  Some  stated  that 
the  time  limit  for  a  reconsideration  elapsed  before  they  received  organization-wide  information 
on  ratings  and  payouts  --  information  that  would  give  perspective  to  their  individual  ratings  and 
help  them  assess  whether  to  request  reconsideration.  Other  employees  simply  did  not 
understand  how  to  request  a  reconsideration.  Senior  leaders  interpreted  the  low  number  of 
reconsideration  requests  to  indicate  employee  satisfaction  with  ratings  and  payouts.  However, 
they  also  acknowledged  a  lack  of  guidance  as  a  drawback  to  a  well-conducted  reconsideration 
process. 

3.3.3.  Themes  Related  to  KPP  5:  Supporting  Infrastructure 

Theme  11.  Difficulty  in  Writing  Performance  Objectives  and  Assessing  Performance 

Supervisors  and  employees  reported  difficulty  writing  SMART  performance  objectives. 
Confusion  about  using  contributing  factors  and  weighting  was  primarily  attributed  to  a  lack  of 
guidance.  Organizations  with  more  experience  under  NSPS,  however,  tended  to  report  less 
trouble  with  writing  objectives. 

Theme  12.  Workforce  Praises  the  Mock  Pay  Pool  but  Identifies  Training  Weaknesses 

Leadership  and  the  workforce  received  different  levels  of  NSPS  training,  with  differing  levels  of 
effectiveness.  The  lag  time  between  training  and  the  actual  use  of  skills  was  sometimes 
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problematic,  and  performance  objective  training  was  often  cited  as  ineffective.  Supervisors 
consistently  praised  mock  pay  pools  as  helpful.  Some  "soft  skill"  instruction  and  contractor- 
supported  training  proved  unevenly  effective. 

3.3.4.  Additional  Themes 

Theme  13.  Changing  Policies  and  Communications  Processes  Made  It  Difficult  To  Understand 

NSPS 

Shifting  policies  resulting  from  NSPS  startup  and  the  2007  change  to  Element  2  complicated 
implementation.  Frequent  NSPS-related  e-mails  from  implementation  teams  to  the  workforce 
led  to  information  overload  in  some  organizations  and  frustrated  a  comprehensive 
understanding.  Among  some  employees,  there  is  awareness  that  NSPS  is  still  taking  root  and 
that  it  will  require  time,  effort,  training,  and  leadership  to  fully  reach  its  promised  potential  and 
for  managers  to  become  fluent  in  its  application.  However,  other  employees  and  managers 
were  pessimistic  that  NSPS  could  succeed. 

Theme  14.  Perceptions  of  IT  Support  Tools 

Most  employees  and  supervisors  found  the  PAA  difficult  and  time-intensive  to  use.  Their 
reasons  included  frequent  inability  to  access  the  tool;  a  non-user  friendly  interface,  including 
cumbersome,  non-intuitive  navigation;  confusing  screen  layouts;  loss  of  data;  CAC  card  usage 
restriction;  lack  of  remote  connectivity  capability;  and  access  limited  to  either  the  employee  or 
the  supervisor  at  any  one  time.  A  lack  of  hands-on  training  and  outdated  or  difficult-to-use  user 
guides  further  complicated  PAA  usage.  Other  IT  support  tools,  such  as  the  Readiness  Tool  and 
the  Compensation  Workbench,  were  more  specialized  and  served  smaller  groups  of  users. 
These  tools  were  generally  reviewed  favorably. 

Theme  15.  NSPS  Performance  Management  System  Demands  Significant  Organizational 

Resources 

The  time  and  administration  requirements  of  the  NSPS  performance  management  system  are 
key  concerns  of  leadership  and  the  workforce.  These  requirements  are  further  complicated  by  a 
wartime  operations  tempo,  staff  shortages  and  transitions,  and  other  non-NSPS  job  demands. 
Many  questioned  whether  NSPS  warrants  the  current  investment  of  time  and  manpower  for 
performance  planning,  reviews,  assessments,  panels,  and  documentation,  or  whether  time 
requirements  are  sustainable  for  the  long  term.  Time  demands  were  also  increased  by  the 
workforce's  difficulty  in  using  the  PAA.  Senior  officials  said  it  was  too  soon  to  tell  whether  there 
are  offsetting  benefits  to  the  increase  in  required  time. 

Theme  16.  Expected  Long-Term  Effects 

There  are  diverse  perspectives  on  the  outlook  for  NSPS.  A  lack  of  clarity  for  career  progression, 
uncertain  pay  raises,  the  option  to  move  to  non-NSPS  organizations,  supervisors'  perceived 
shortcomings,  and  a  more  competitive  workplace  climate  are  predicted  to  impact  negatively 
upon  retention  and  job  satisfaction.  Some  cite  competing  pay  systems,  ops  tempo,  and  a 
change  of  administration  as  challenges  to  the  success  of  NSPS.  Others  offered  that  additional 
guidance,  leadership,  and  training  would  help  NSPS  realize  its  full  effectiveness.  Senior  leaders 
were  cautiously  optimistic  that  NSPS  can  eliminate  poor  performers  and  generally  wished  to 
pursue  NSPS  to  maturity,  but  also  indicated  that  it  is  too  early  to  evaluate  the  effects  of  NSPS 
on  individual  and  organizational  performance. 
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3.4.  Contextual  Information 

During  the  site  visits,  participants  discussed  several  factors  that  were  not  directly  part  of  NSPS 
but  were  believed  to  have  an  influence  on  NSPS  implementation  and  success.  These  contextual 
factors  include: 

■  GS  as  a  competing  system 

■  The  NDAA  pay  pool  funding  change 

■  High  operations  tempo  (wartime  ops  tempo) 

■  Personnel  center  backlogs 

■  Residual  GS  requirements 

■  Military  supervisors 

■  Supervisor  capabilities 

These  contextual  issues  are  explained  in  detail  below. 

3.4.1.  GS  as  a  Competing  System 

The  continued  existence  of  the  General  Schedule  (GS)  in  DoD  is  seen  as  complicating 
acceptance  of  NSPS.  Employees  expressed  concerns  related  to  differences  in  career 
opportunities  and  pay  that  result  from  having  parallel  systems. 

3.4.2.  The  NDAA  Pay  Pool  Funding  Change 

The  2008  National  Defense  Authorization  Act  (NDAA)  changed  Element  2  pay  pool  funding  after 
many  organizations  had  distributed  their  first  cycle  payout  results.  This  change  confused  many 
employees,  raising  concerns  about  the  credibility  of  NSPS. 

3.4.3.  High  Operations  Tempo  (Wartime  Ops  Tempo) 

Senior  leaders,  supervisors,  and  employees  cited  the  current  high  operations  tempo  as  a 
significant  challenge  during  peak  periods  of  NSPS  administration,  such  as  at  deadlines  for 
performance  appraisals  and  during  pay  pool  panel  meetings.  For  example,  pay  pool  panel 
members  noted  that  they  -  as  senior  leadership  -  may  be  diverted  from  critical  mission  tasks  to 
sit  on  panels,  sometimes  for  weeks  at  a  time.  Supervisors  indicated  that  it  was  challenging  to 
balance  time-sensitive  mission  demands  with  the  time-sensitive  demands  to  write  employee 
performance  appraisals  and  defend  them  before  the  pay  pool  panel. 

3.4.4.  Person ne I  Center  Backlogs 

Personnel  action  backlogs  at  a  component  personnel  center  were  seen  to  adversely  impact 
NSPS  hiring  and  reassignment  flexibilities.  Some  staff  reported  that  it  took  up  to  eight  months 
for  a  reassignment  within  an  NSPS  organization  at  affected  sites. 

3.4.5.  Residual  GS  Requirements 

Some  organizations  retain  command-directed  GS  requirements  that  complicate  NSPS  processes 
and  confuse  perceptions  of  NSPS  as  a  flexible  system.  In  Air  Force  organizations,  NSPS 
employees  who  seek  advancement  are  required  to  submit,  in  addition  to  the  NSPS  rating  of 
record,  the  GS  form  for  promotion  with  the  supervisor's  assessment  on  the  GS  9x9  scale.  This 
GS  form  is  then  used  by  hiring  officials  to  make  a  selection.  Another  organization  reportedly 
requires  panel  selection  boards  to  make  in-house  reassignments  rather  than  rely  on  NSPS 
flexibility  allowing  supervisors  to  select  and  make  reassignments.  Retaining  GS  concepts  such  as 
these  potentially  serves  to  reduce  NSPS  flexibilities. 
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3.4. 6.  Military  Supervisors 

Military  supervisors,  who  rotate  frequently  and  may  be  new  to  an  organization,  rate  both  NPSP 
and  GS  employees.  If  they  rotate  in  the  middle  of  a  rating  cycle,  they  may  lack  knowledge  of 
employee  performance.  Military  supervisors  may  also  lack  training  on  NSPS,  including 
understanding  of  the  importance  of  the  performance  plans. 

3.4.7.  Supervisor  Capabilities 

NSPS  places  a  significant  amount  of  responsibility  upon  supervisors  to  mentor,  coach,  and 
advise  employees.  Some  employees  believe  that  their  supervisors'  lack  of  writing  skills  may 
harm  their  ratings,  payouts,  and  careers. 

The  following  section  describes  these  themes  in  greater  detail. 

3.5.  Themes  and  Findings 

3.5.1.  Themes  Related  to  KPP 1:  High-Performing  Workforce  and  Management 
This  KPP  relates  to  issues  concerning: 

■  Relationship  between  performance,  performance  ratings,  and  performance  payouts 

■  Effects  of  NSPS'  flexibilities  in  recruitment  and  retention 

■  Extent  to  which  useful  performance  feedback  is  provided 

■  Alignment  of  individual  and  organizational  goals 

■  Extent  to  which  NSPS  supports  employee  engagement 

Theme  1.  Link  Between  Performance  and  Pay  Is  Moderated  by  Several  Factors 

Link  between  Performance  and  Pay 

There  was  a  mixed  assessment  of  the  link  between  performance  and  pay  under  NSPS.  Some 
employees  believed  they  saw  some  level  of  pay-for- perform  a  nee,  while  other  employees  were 
uncertain  because  marginal  performers  were  not  rated  as  such  and  high  performers  were  not 
recognized  and  rewarded  as  they  felt  they  should  have  been.  Other  employees  expressed  the 
opinion  that  NSPS  is  not  pay-for-performance,  citing  the  following  as  reasons:  a  perceived 
forced  distribution  on  ratings;  reliance  on  the  quality  of  supervisor  write-ups  versus  actual 
employee  accomplishments;  perceived  inconsistent  willingness  of  supervisors  to  defend  their 
recommended  ratings;  the  quality  of  the  employee-supervisor  working  relationship;  budget 
constraints;  pay  pool  rounding  rules;  and  supervisors'  reluctance  to  accurately  rate  marginal 
performers. 

"The  2008  NDAA  changes  to  the  NSPS  will  weaken  what  DoD  envisions  for  this  system.  The  loss  of  the 
workforce  shaping  feature  has  left  us  in  a  split  environment  of  rewarding  employees  based  on 
performance,  and  retaining  based  on  longevity.  It  sends  the  wrong  message." 

Implementation  Team  Member 

"When  I  first  came  onboard  under  GS,  the  rating  was  either  acceptable  or  unacceptable.  I  preferred  to 

be  rated  based  on  performance NSPS  is  great  because  it  separates  the  good  from  the  bad  and 

highlights  valued  performers.  For  me,  it  has  given  me  the  motivation  to  say  that  this  is  my  chance  to 

show  that  I  am  not  just  'acceptable.'" 

"Everything  revolves  around  pay,  but  DoD  really  wanted  to  improve  performance.  This  is  lost.  It  hasn't 

improved  performance."  [Agreement;  heads  nod.] 

Employees 

Focus  on  Rating  Level  3 

Some  employees  cited  that  training  and  leadership  had  emphasized  that  the  majority  of 
employees  would  be  rated  3.  They  claimed  that  this  allowed  supervisors  to  assign  a 
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preponderance  of  3  ratings,  without  working  to  determine  whose  performance  may  have  been 
superior. 

"Our  belief  is  that  the  system  wants  to  put  everyone  at  the  center  of  mass,  and  only  the  glowing  will 

stand  out." 

Supervisor 

Differentiating  High  and  Marginal  Performers 

Some  employees,  supervisors,  and  commanders  cited  the  reluctance  to  give  a  2  rating  to 
marginal  performers.  Other  supervisors  worried  that  higher  performers  who  were  still  rated  3 
were  not  better  rewarded. 

"I  hoped  NSPS  would  give  those  not  performing  well  (the  motivation  to)  perform  better,  but  that's  not 
happening  either.  They  may  not  see  the  increase,  but  nothing  bad  happens  to  them,  either." 

Employee 

"What  would  be  the  consequences  of  giving  non-contributors  a  2?  Institutionally  we're  not  used  to 

doing  that." 

"Top  performers  are  being  adequately  compensated.  It  is  fuzzier  in  the  middle.  There  needs  to  be  a 

better  way  to  make  distinctions  in  middle  performers." 

Supervisors 

"More  people  probably  should  have  gotten  2s." 

Pay  Pool  Panel  Member 

"(T)here  is  potential  for  higher  performers  to  be  better  rewarded.  It  is  very  difficult  for  lower 

performers  to  be  compensated  well." 

"We  have  not  used  level  2  ratings  to  the  extent  that  we  should  use  it.  We’re  not  emotionally  there 

yet." 

Commanders 

Importance  of  Job  Visibility 

Some  employees  were  under  the  impression  that  more  mission-central  or  higher-level  jobs  are 
more  likely  to  receive  true  pay-for- perform  a  nee  than  those  jobs  not  central  to  the  mission  or 
with  less  responsibility.  Supervisors  were  concerned  that  more  routine  or  less  mission  central 
jobs  may  not  get  the  higher  ratings  merited  by  employees'  outstanding  performance. 

"There's  an  institutional  bias  or  preference  to  certain  functions.  Once  you're  in  a  certain  area,  you're 
branded.  We  were  told  that  really  high  payouts  were  skewed  to  more  'central'  functions.  How  does  a 
system  reward  people  who  are  deserving  and  doesn't  penalize  other  people  who  are  doing  well  in  'less 

central'  functions  ?" 

Employee 

"You  have  to  show  how  they  were  able  to  show  their  technical  expertise  outside  the  organization,  so 
we  have  to  use  terms  like  [Major  Command],  DoD.  If  I  can  show  we  are  going  above  and  beyond  (I  will 
get  higher  ratings),  but  how  do  you  do  that  at  the  child  care  center  or  the  fitness  center?  But  that's  as 

important  because  they  allow  us  to  do  our  work." 

Supervisor 

Importance  of  Writing  Skill 

Many  employees  and  supervisors  asserted  that  the  quality  of  writing  is  a  key  factor  in 
determining  the  extent  of  pay  for  performance.  Some  managers  observed  that  writing  quality 
was  a  significant  contributor  to  how  the  ratings  were  substantiated  and  recommended 
improvement  across  the  board  in  the  quality  of  assessments. 

"The  whole  thing  is  so  writing  skill  based,  that  you  are  going  to  be  unable  to  train  people  to  write  to 
this  level...  it  limits  a  whole  group  of  people  whose  supervisors  can't  write  well." 
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"If  you  can  write  well,  you  can  make  a  clock  puncher  look  like  a  water  walker." 

Employees 

"They  kept  saying  in  the  training  'finally,  a  system  where  you  get  paid  for  your  performance'...  it's  not 

that  at  all...  it's  how  well  you  write." 

Supervisor 

"When  NSPS  was  first  marketed,  people  said,  'Oh,  it  depends  on  how  good  a  writer  your  supervisor  is.' 

We  said  'no,'  but  then  found  that  all  you  had  in  the  Mock  (pay  pool)  was  the  written  word,  so  it  was 
sort  of  true.  We  communicated  to  employees  that  while  it  is  not  a  writing  exercise,  that  is  all  we  have, 
so  it  is  important.  We  ALL  have  to  get  better  at  writing  performance  statements." 

"The  eloquence  of  the  writing  is  important.  You  need  to  focus  reader  into  what  you  want  to  see.  The 
big  thing  is  the  'so  what'  -  what  did  that  contribute,  what  did  it  advance;  what  was  the  impact?  It's 
not  just  what  you  did.  There  is  a  perception  that  this  is  a  writing  competition,  but  that  is  wrong.  That 

said,  it  is  also  not  entirely  objective." 

Pay  Pool  Panel  Members 

"I  think  the  ability  to  write  is  the  issue  here....  If  supervisors  could  not  express  the  performance 
articulately,  you  could  easily  have  a  true  5  who's  a  3  because  the  rules  of  engagement  make  that 
person  a  3....  I  just  don't  think  that's  fair ...  could  be  2  or  3  shares;  that's  a  big  impact  on  their  life." 

Implementation  Team  Member 

Budget  and  Policy  Seen  as  Constraints  on  Pay-Performance  Link 

Some  organizations  have  established  business  rules  stating  that  a  2  rating  is  an  acceptable  level 
of  performance  instead  of  a  3.  Employees  fear  that  this  policy  could  have  an  effect  outside  the 
organization  on  rating  perceptions  and  internally  on  pay  increases.  Some  supervisors  asserted 
that  assigning  whole  shares  reduces  the  relationship  between  performance  and  pay  by 
providing  identical  outcomes  to  individuals  with  different  ratings.  Implementation  teams' 
concerns  addressed  payouts  based  on  a  percentage  of  salary,  which  rewarded  more  senior 
employees  and  disadvantaged  more  junior  employees.  They  argued  that  this  strategy  is  at  odds 
with  pay  for  performance  because  it  tends  to  reward  longevity  (i.e.,  those  who  have  been 
around  longer  tend  to  be  higher  in  the  pay  band  and  therefore  get  bigger  raises,  even  though 
more  junior  employees  who  are  lower  in  the  pay  band  might  have  performed  better). 

"Payouts  are  driven  by  the  Agency's  budget  and  there  are  a  lot  differences  in  the  size  of  the  budgets." 

Employee 


Theme  2.  Employees'  Negative  Perception  of  a  3  Rating 

The  Meaning  of  a  3  Rating  and  How  To  Exceed  It 

Employees  and  supervisors  exhibited  a  strong  reluctance  to  accept  "valued  performer"  -  or  3  - 
as  a  good  rating.  Many  stated  they  cannot  articulate  what  they  must  do  to  exceed  a  3  rating, 
and  that  they  have  difficulty  in  relating  performance  indicators  to  writing  their  objectives  and 
self-assessments.  Pay  pool  panel  members  generally  agreed  that  some  supervisors  do  not  know 
how  to  use  performance  indicators  in  writing  employee  assessments.  Employees  also  perceived 
that  lower-responsibility  jobs  will  not  be  able  to  exceed  a  3  rating.  Employees  generally  agreed 
that  performance  improvement  guidance  is  not  clear  in  NSPS. 

"It  would  be  nice  if  they  published  some  sort  of  analysis.  Here  are  (the  organization's)  3s,  here's  the  4s, 

and  the  5s.  What  qualities  separated  these?  What  were  they  looking  for?" 

Employee 

"Tell  us  what  a  5  looks  like  so  we  can  gauge.  There  was  nothing  like  this  in  training." 

Supervisor 

"The  pay  pool  panel  administrator  would  contact  the  rater  with  questions  saying  that  the  assessment 
does  not  support  the  recommended  ratings  and  that  they  have  an  opportunity  to  rectify  the  ratings. 
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This  happened  for  70%  to  80%  of  ratings.  Supervisors  did  not  understand  what  it  took  to  get  a  4  or  5. 
They  would  write  things  like  ' employee  is  an  excellent  performer,  customers  love  her,  or  very  smart 

technically ;'  which  does  not  get  you  a  5." 

Pay  Pool  Panel  Member 

Impact  of  a  3  Rating  on  Career 

A  number  of  employees  expressed  concern  about  negative  perception  by  non-NSPS 
organizations  of  NSPS  3  ratings  and  potential  impact  on  inter-system  transfers  (NSPS  to  GS/APS) 
or  transfers  to  other  NSPS  commands  that  may  have  different  rating  standards.  Pay  pool  panel 
members  noted  supervisors'  concerns  about  3  ratings'  reception  in  other  organizations. 

"A  lot  of  us  had  a  problem  with  the  fact  that  ‘3’  was  outstanding.  The  3.0  or  the  C  is  average.  How  is  3 
outstanding  on  a  scale  of  1-5?  '4'  or  '5'  is  outstanding.  If  you  move  on,  how  are  they  going  to  look  at 

that  scale?" 

Employee 

"Some  supervisors  can't  get  past  the  fact  that  level  3  is  a  'valued  performer'  and  are  concerned  about 

how  a  3  will  be  looked  at  if  they  move  to  a  non-NSPS  organization." 

Pay  Pool  Panel  Member 

Potential  for  Gamesmanship 

Some  employees  were  said  to  be  gaming  the  system  to  get  higher  ratings.  Gaming  techniques 
included  writing  performance  objectives  below  the  expected  level  of  work  to  show  the 
employee  exceeded  them  or  taking  on  additional  duties  for  recognition  of  extra  work  at  the 
cost  of  neglecting  primary  tasks.  Gamesmanship  at  the  workforce  level  may  indicate  an 
incomplete  understanding  of  the  use  of  performance  indicators  to  rate  objectives.  If 
undetected,  gamesmanship  poses  an  obstacle  to  fairness  and  system  effectiveness. 

"I  put  minimal  stuff  in  my  actual  objective  so  that  I  could  put  extra  things  in  my  self-assessment  to 
make  it  look  like  I  did  more  than  the  objective.  That's  what  the  instructor  told  us  to  do." 

Employee 

"There  is  concern  about  the  impact  of  being  honest  in  the  system.  If  people  are  honest,  we  will  stay 
with  3s  and  the  employees  will  get  less  money.  They  are  in  competition  for  jobs  with  other  bases  (that) 

may  not  be  honest  in  their  ratings.  If  this  happens,  people  will  lose  faith." 

Pay  Pool  Panel  Member 

Impact  of  a  "Valued  Performer"  Rating  on  Employee  Morale 

Some  employees  voiced  the  opinion  that  ratings,  as  a  measure  of  worth,  were  more  important 
than  money.  A  few  other  employees,  notably  long-term  civil  servants,  indicated  a  rating  does 
not  serve  as  a  motivator  for  future  professional  performance  but  that  inherent  work  ethic  does. 
The  overall  impact  of  a  3  rating,  cited  by  many  in  the  workforce,  is  a  decrease  in  morale  and  a 
threat  to  their  self-esteem  and  identity  as  professionals  and  previous  high-mark  performers. 

"In  the  old  system,  almost  all  people  received  the  maximum  rating.  In  the  new  system,  most  people 
receive  a  3.  Only  a  few  people  get  1  or  2.  This  really  creates  a  self-esteem  issue.  The  'valued  employee' 
wording  for  a  3  is  nice,  but  let's  be  honest,  this  is  still  a  bump  down."  [Many  agree.] 

"When  I  got  a  3, 1  was  put  into  the  same  class  of  employees  who  don't  care  or  who  take  leave  all  the 
time  -  the  people  who  call  in  sick  once  a  week.  I  was  a  valued  employee  under  GS  and  I  knew  when  I 

was  getting  my  step  increase.  But,  everyone  is  a  valued  employee  now. " 

"Giving  someone  a  3  means  you're  doing  your  job.  My  work  ethic  motivates  me  to  work.  A3  is  a 
generic  that  only  determines  how  much  money  you  get....  I  don't  mind  the  numbers." 

Employees 

"How  do  I  tell  employees  that  get  8s  and  9s  (under  the  GS  system)  that  a  3  is  as  good?" 

Supervisor 
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"The  differences  in  the  rating  numbers  (e.g.,  3, 4,  5)  mean  a  lot  to  some  people.  People  feel  like  a  3  is  a 
'C.'  It's  hard  for  them  to  understand  a  3  is  a  valued  employee  that  is  doing  what  is  expected." 

PRA 

"I  will  look  at  4s  and  5s  differently  than  before  and  will  stress  this  to  leadership.  To  get  a  5,  you  must 

be  a  change  agent.  Working  long  hours  is  not  enough  to  get  a  5." 

Commander 

Theme  3.  Effects  on  Recruitment  and  Retention 

Use  of  NSPS  Flexibilities 

There  were  mixed  assessments  of  the  impact  of  NSPS  on  recruitment,  hiring,  and  retention. 
Organizational  policy  is  seen  to  sometimes  allow  as  well  as  to  sometimes  limit  NSPS  flexibilities. 
Some  employees  appreciated  the  flexibility  and  ability  to  grow  and  increase  salary  within  a  pay 
band  system.  Some  supervisors  asserted  that  NSPS  is  not  meeting  hiring  flexibility  expectations 
because  of  a  limited  number  of  pay  bands  and  the  superimposition  of  GS-based  "pay  lanes" 
upon  NSPS  pay  bands;  others  expressed  appreciation  for  the  increased  flexibility  in  making 
assignments.  A  few  supervisors  were  concerned  that  less-qualified  personnel  may  use  broad 
pay  bands  to  seek  supervisory  positions  as  a  means  to  advance  and  that  this  could  impact 
negatively  upon  the  organization.  Pay  pool  panel  members  and  managers  identified  the  ability 
to  negotiate  starting  salaries  as  a  positive  aspect.  PRAs  saw  money  as  a  motivator  for  better 
performance.  A  few  employees  opined  that  supervisors'  unfamiliarity  with  pay-setting,  a  new 
concept  under  NSPS,  hinders  in-system  recruiting  as  potential  outside  hires  negotiate  higher 
starting  salaries  that  were  not  available  to  current  employees.  Some  directors  worried  about 
the  impact  of  negotiable  pay  on  their  budget. 

"If  someone  is  a  valued  employee,  you  don't  have  to  wait  to  give  that  person  a  raise.  This  is  a  positive 

for  employees  who  really  deserve  more  money." 

"We  have  had  a  lot  of  jobs  declined  because  we  cannot  set  the  pay  correctly  under  NSPS." 

"Jobs  are  never  advertised.  Hidden  promotions.  (There's)  no  opportunity  to  compete,  (and)  hires  from 

outside  could  get  higher  increases." 

Employees 

“We’ve  had  to  shuffle  people  around.  Under  NSPS  it's  been  so  much  easier  -  it's  been  great.  A 
negative,  however,  is  that  there  are  a  limited  number  of  pay  bands.  A  lot  of  my  people  are  at  the  top 
of  their  band  and  there  is  no  where  else  for  them  to  go.  They  can  get  bonus  but  they  are  limited  in 

what  they  can  get  in  salary." 

"We  have  dealt  with  it  when  we  try  to  hire,  but  we  have  been  locally  directed  to  have  pay  lanes  which 

mirror  GS.  So  there  is  no  flexibility  to  advertise  jobs  using  the  broad  bands." 

Supervisors 

"If  the  department  is  going  to  hold  each  organization  accountable  for  executing  its  budget,  then  why 
the  limitation  of5%?  In  some  cases,  we  become  non-competitive  with  external  organizations  when  we 
are  limited.  To  recruit  good  talent,  sometimes  you  have  to  make  an  offer  that  exceeds  the  cap.  So 
NSPS  organizations  are  at  a  significant  disadvantage  when  compared  to  GS  organizations  in  recruiting 

and  retaining  high-quality  positions." 

Pay  Pool  Panel  Member 

"We  did  not  use  control  points.  (Our  organization)  didn't  feel  as  though  we  were  far  enough  along  to 
consider  control  points.  They  didn't  want  to  use  them  in  the  first  year.  We  did  not  use  EPI  or  OAR 
either.  We  wanted  to  make  the  most  out  of  the  pay  pool  process  in  the  first  year  and  wanted  to  make 
sure  we  had  consistency  across  the  pay  pools.  We  were  early  on  in  the  process  and  wanted  to 
minimize  the  variability.  We  also  didn't  use  weighting  for  the  same  reasons.  We  kept  things  the  same 

for  this  year  as  well." 

PRA 
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Recruiting  the  Highly  Qualified 

Pay  Pool  Panel  members  and  managers  identified  recruiting  supervisors,  pay  band  caps,  hiring 
delays  due  to  pay  negotiations,  and  tailoring  vacancy  announcements  within  broad  pay  bands 
or  for  specialties  as  challenges  to  recruiting.  Some  employees  believed  NSPS  flexibilities  benefit 
outside  hires  more  than  current  employees  because  outside  hires  have  the  ability  to  negotiate 
higher  starting  salaries.  Employees  pointed  out  that  they  cannot  search  for  positions  at  specific 
pay/experience  level  as  they  did  under  the  GS  system.  PRAs  cited  NSPS  allows  more  flexibility  in 
recruiting  and  retaining  employees  with  higher-level  expertise. 

"It's  more  beneficial  for  someone  off  the  street.  Managers  can  set  their  pay.  If  you're  taking  a 
reassignment,  you  have  taken  on  more  responsibilities  for  less  or  the  same  pay  because  they  don't 

have  to  give  you  the  5%." 

Employee 

"I've  seen  people  turn  down,  won't  go  into  NSPS,  because  they  won't  get  paid  for  more 

responsibility." 

Supervisor 

"Some  make  the  connection  that  if  I  take  on  higher  level  duties  it  will  give  me  more  opportunities  (for 
reassignment).  But  for  some  that  5%  is  not  worth  it.  Supervision  under  NSPS  is  not  for  the  weak." 

Pay  Pool  Panel  Member 

"One  of  the  things  of  NSPS  is  to  allow  us  to  be  more  competitive,  recruit  and  retain  our  higher-skilled 
employees.  It's  been  a  huge  challenge.  That's  a  positive  of  NSPS,  getting  away  from  the  186K  cap.  We 
had  the  opportunity  to  compensate  to  what  is  being  offered  (elsewhere).  This  has  helped  with 

recruiting  and  retention." 

"Having  more  than  one  system  on  the  base  also  makes  pay  setting  difficult.  You  need  some 

consistency  across  the  board  in  how  you  do  this." 

PRAs 

Effects  on  Retention 

Employees  cited  the  following  as  issues  and  challenges  to  retention:  lack  of  clarity  for 
progression;  the  5%  limit  on  salary  increases  for  reassignments  with  more  demands;  declining 
morale  and  job  satisfaction;  a  perception  of  making  less  money  than  GS  employees;  lack  of 
supervisor  knowledge  and  skills  to  properly  ensure  good  ratings  for  the  employee;  and  new 
component  policies  encouraging  employees  to  move  frequently  to  positions  in  new 
organizations  or  to  a  supervisor  position  in  a  lower  pay  band.  Supervisors  and  employees  saw 
turnover  among  first-line  supervisors  occurring  because  of  the  increased  workload  under  NSPS. 
Pay  Pool  Panel  members  and  managers  identified  retaining  supervisors,  perceived  pay 
inequities  between  current  and 
new  employees,  top-of-pay-band 
pay  caps,  tailoring  vacancy 
announcements  within  broad  pay 
bands,  and  ratings  inflation  as 
challenges  to  retention.  PRAs  and 
Pay  Pool  Panels  saw  "system 
shopping"  between  NSPS  and  GS 
to  gain  salary  as  an  issue  in 
retention. 

"One  of  the  issues  we  have  is  that  we  still  need  to  manage  the  payroll.  An  additional  burden  is  that 
you  have  people  in  pay  bands,  but  you  only  have  so  much  money.  The  organization  must  look  out  for 

the  individual,  but  you  can't  promote  more  than  you  have  money." 

Employee 


Pay  Pool  Panel/Senior  Manager  Perspectives  on  Recruiting,  Hiring,  and 

Retention 

Positives 

(1)  The  ability  to  negotiate  starting  salaries 

(2)  That  new  hires  can  start  at  a  level  commensurate  with  their  experience 

(vs.  default  of  starting  at  the  bottom) 

Challenges 

(1)  Recruiting  and  retaining  supervisors 

(2)  Perceived  pay  inequities  between  current  and  new  employees 

(3)  Time  delays  due  to  pay  negotiations 

(4)  Tailoring  vacancy  announcements  within  broad  pay  bands 
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"There  are  a  number  of  senior  people  who  have  retired  because  of  NSPS.  They  did  so  right  before  we 
converted  because  there  is  too  much  work  load  in  NSPS  and  a  larger  administrative  burden." 

Supervisor 

"Part  of  retention  is  related  to  keeping  ratings  inflation  down.  The  more  shares  there  are,  the  lower 
the  value  of  each  share.  Need  to  keep  ratings  in  check  so  that  people  who  do  well  get  a  really  good 

pay  raise.  Really  important  to  keep  ratings  inflation  down." 

"(I)  have  a  concern  about  system  shopping.  As  it  stands  now,  if  you  go  from  GS  to  NSPS  you  get  3-5%. 
If  you  get  out  and  go  back  to  GS,  you  go  in  at  a  comparable  level  (the  increase)  and  then  may  want  to 

go  back  to  NSPS.  You  can  shop  around  until  you  get  pay  increases." 

Pay  Pool  Panel  Members 

"In  the  past,  working  supervisors  did  more  work  than  supervise,  and  now  they  have  to  supervise 

more.  In  the  old  system  we  never  asked  them  to  do  this." 

"In  exit  interviews,  some  supervisors  say  they  are  leaving  because  they  don't  want  to  spend  time  on 

NSPS." 

Implementation  Team  Members 

Effects  of  Conversion 

The  conversion  from  the  GS  system  to  pay  bands  placed  many  supervisors  in  Pay  Band  2  while, 
in  some  cases,  placing  their  technical  employees  in  Pay  Band  3.  Many  employees  found  this 
structure  confusing  and  dissatisfying.  Some  employees  complained  that  distinctions  of  seniority 
were  lost  with  the  inclusion  of  senior  GS  grades  into  supervisory  Pay  Band  2. 

"A  lot  of  the  technical  folks  had  bitterness  about  pay  banding  because  they  were  13s  and  really 
experts  and  they  were  not  recognized  as  experts.  They  were  lumped  in  with  Journeymen." 

Employee 

"I  had  a  YC2,  wanted  to  encourage  him  to  sign  up  for  Air  War  College.  But  only  GS-14s/15s  were 
allowed  on  the  website.  So  he  was  denied.  So  translation  is  needed.  It  is  going  to  be  real  confusing." 

Supervisor 

"Broad  bands  make  it  so  it's  worth  more  to  get  out  of  the  supervisor  pay  band  and  get  into  the 
technical  band.  We  hear  a  lot  of  supervisors  want  to  get  out  to  the  technical  band  because 

[supervision]  is  not  worth  their  time  and  effort." 

Implementation  Team  Member 

Theme  4.  Confusion  about  Advancement  within  and  between  Pay  Bands 

Advancement  within  an  NSPS  Pay  Band 

As  expected  so  recently  after  a  major  change,  many  employees  indicate  residual  connection  to 
the  previous  system,  evidenced  by  continued  references  to  GS  terms  such  as  grades  and 
promotions  instead  of  the  NSPS  concepts  of  advancement  and  pay-for- perform  a  nee  increases. 
With  the  new  pay  bands  versus  the  familiar  grades  and  steps,  many  employees  did  not 
understand  how  they  would  advance  within 
NSPS.  While  some  employees  recognized  a 
greater  mobility  and  opportunity  to  grow, 
others  were  confused  about  what  happens 
upon  reaching  the  top  of  the  pay  band.  A 
significant  number  of  employees  were 
confused  about  their  billet  and  position 
expectations  after  the  conversion  into  the 
large  Pay  Band  2.  A  number  of  employees 
cited  a  lack  of  guidance  explaining 
advancement  in  NSPS  or  cited  the  five 
percent  pay  increase  limit  as  a  discouraging  factor  in  seeking  a  reassignment.  Pay  pool  panel 


"The  one  aspect  that  benefitted  me  was  the  pay  band 
itself.  Combining  GS  14  and  15  into  a  pay  band  allowed  me 
to  take  my  current  job  without  taking  a  grade  cut.  Pay 
band  3  allowed  me  to  move  without  changing  my 
designator.  Pay  bands  allow  movement  between  and 
among  jobs  without  considering  time  in  grade  and  those 
types  of  factors." 

"For  me  it's  the  opposite  ...  I  can  stay  where  I  am  and  move 
up  without  changing  jobs." 

— Employees ,  discussing  the  positive  career  advancement 

aspects  of  pay  bands 
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members  cited  the  increased  supervisory  duties  under  NSPS.  This  group  also  noted  it  was 
difficult  to  determine  which  changes  in  job  duties  merit  a  reassignment  within  a  band.  While 
some  employees  asserted  that  taking  on  new  duties  no  longer  makes  one  eligible  for 
"promotion,"  others  mark  "advancement"  by  being  labeled  as  "team  leaders."  A  few  employees 
purported  that  career  progression  does  not  exist  in  NSPS.  Some  implementation  team 
members  noted  a  tendency  to  retain  GS  concepts  as  overlay  for  NSPS  concepts  and  stressed  the 
need  to  move  away  from  those  concepts.  In  a  system  that  requires  specific  description  of 
accomplishments,  a  small  number  of  employees  working  classified  projects  said  it  was 
impossible  to  provide  detail  about  what  they  did  to  meet  their  objectives  and,  therefore, 
affects  their  opportunity  for  performance-based  pay  increases. 

"Seems  like  there  is  a  lot  of  ability  to  grow,  maybe  faster  than  you  can  under  GS.  It  rewards  the  hard 
chargers,  which  is  good.  But  I  don't  know  what  to  expect  when  you  get  to  the  top  of  the  band." 

"What  does  YA-2  mean?  Are  they  a  (GS-)  7  or  (GS-)  13?  Bands  are  too  broad  to  define  what  that 

person  is  supposed  to  accomplish. " 

Employees 

"It  now  de-motivates  people  by  putting  us  in  this  big  pay  band.  Where's  the  motivation  for  me  to  get 
to  the  13  level  that  was  there  before?  Those  people  also  are  in  the  same  pay  band  and  have  a  ceiling." 

Supervisor 

"There  aren't  promotion  opportunities  for  people  in  (a)  very  wide  pay  band  (Band  2).  This  has  been 
discouraging  to  people  on  the  end  of  that.  Moving  jobs  will  get  you  through  pay  bands  much  faster 

than  annual  salary  increase.  I  have  concerns  about  this." 

Pay  Pool  Panel  Member 

"The  YA2  band  is  a  very  large  band....  Everyone  who  goes  into  that  band  thinks  they  will  eventually  be 
a  13.  But  the  level  of  work  in  the  band  will  never  allow  for  that.  This  band  needs  to  be  made  smaller." 

Implementation  Team  Member 

Intra-  and  Inter-System  Transfers 

A  majority  of  employees  called  for  one  personnel  compensation  system  for  all  of  DoD.  Many 
employees  were  confused  about  how  to  reconcile  NSPS  pay  bands  with  GS  grades  during  the 
job  application  process  as  they  seek  career  advancement. 

"What  happens  if  I  want  to  go  to  another  agency?  Are  they  going  to  know  what  YA-2  means?" 

Employee 

Theme  5.  Payout  Amounts  Were  Generally  Liked,  but  Inclusion  of  the  Government-Wide  Pay 

Increase.  Effects  on  Retirement,  and  GS-Equivalence  Are  Issues 

Satisfaction  with  Rating  and  Payout 

Many  employees  were  not  satisfied  with  their  ratings  but  were  generally  satisfied  with  their 
payouts.  Supervisors  agreed,  stating  many  employees  were  content  with  their  payouts,  but  that 
they  contested  rounding  rules.  Some  employees  found  pay  to  be  the  most  important  aspect  of 
pay  pool  panel  decisions,  and  others  thought  rating  and  recognition  were  most  important. 

"Money  wise  I  did  well,  but  I  got  a  3." 

"It  was  close  to  how  I  would  have  rated  myself.  But  in  comparison  to  the  rest  of  my  pay  pool,  my 

rating  was  incredibly  low." 

"The  difference  in  pay.  I'm  making  $6,000  less  than  before." 

"I  saw  improvement  in  pay  as  an  average  rated  employee.  If  you  got  2  or  3  shares,  you  would  do 

better  than  what  you  would  have  gotten  under  GS  system." 

Employees 
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"The  feedback  I  got  from  people  is  that  they  could  live  with  the  rating,  but  it  was  the  payout.  A  2.51 
was  rounded  to  a  level  3  and  got  the  same  payout  as  a  3.00.  We  need  better  differentiation." 

Supervisor 

"I  have  never  heard  of  payouts  as  large  as  we  had." 

Pay  Pool  Panel  Members 

Satisfaction  with  Payout  Policy 

Some  employees  and  supervisors  were  surprised  that  part  of  the  government-wide  pay 
increase  was  included  in  the  pay  pool  budget.  A  number  of  employees  were  concerned  about 
the  effects  of  variable  salary/bonus  payouts  on  their  "retirement  high  3"  as  bonus  does  not 
count  towards  retirement  pay.  There  were  varying  views  about  pay  caps  at  the  tops  of  bands, 
with  some  pleased  with  payout  possibilities  and  some  not.  Some  employees  saw  rounding  rule 
policies  as  disadvantageous  with  a  negative  effect  on  pay.  A  few  employees  were  happy  with 
their  payouts,  but  expressed  worry  about  future  payouts  not  matching  up.  A  few  employees 
who  were  facing  retirement  expressed  dissatisfaction  with  the  bonus  policy  that  requires 
working  until  after  January  1  to  receive  the  bonus.  Employees  and  supervisors  perceived  pay 
differences  between  NSPS  and  GS,  with  some  asserting  GS  has  overall  better  pay. 

"Pay  is  capped  under  NSPS,  but  in  the  GS  system  you  can  still  continue  to  accrue  retirement.  This  is  a 
big  system  inequity.  Also,  if  you  get  your  payout  as  a  bonus  and  not  as  a  salary,  it  does  not  count 

towards  your  retirement.  This  puts  you  at  a  major  disadvantage." 

"I'm  at  the  top  of  the  pay  band,  and  I'll  be  better  off  because  I  can  get  rewards  whereas  before  I  was 

capped  out." 

"Apparently  you  have  to  be  working  in  January  or  February  to  receive  the  cash  award  for  the  previous 
year.  That  affects  when  you  can  retire,  which  seems  unfair  because  it  (the  performance  bonus)  is  for 

past  performance.  You  should  get  it  whether  you  are  here  or  not." 

"We  are  told  that  the  beauty  of  NSPS  is  that  you  can  blow  through  the  pay  cap,  but  you  can't  because 

(the  organization)  has  imposed  its  own  control  points." 

Employees 

"Why  are  we  penalized  and  not  getting  cost  of  living?" 

Supervisor 


Theme  6.  Mixed  Value  of  Interim  Reviews  and  Self-Assessments 

Interim  Reviews 

Many  employees  held  unfavorable  opinions  of  the  interim  review  process.  A  primary  reason  for 
this  was  the  supervisor's  inability  to  offer  concrete  and  specific  measures  of  improvement.  A 
second  major  reason  was  the  difference  between  the  supervisor's  interim  review  assessment 
and  the  pay  pool  panel's  final  rating  of  record.  Some  employees  reported  that  interim  reviews 
seldom  allowed  for  career  path  discussions.  Supervisors  generally  saw  interim  reviews  as 
valuable  as  they  were  more  formal  than  under  the  previous  system,  but  were  concerned  about 
giving  positive  feedback  because  it  might  have  set  an  expectation  for  the  final  rating. 

"I  asked  how  I  could  improve  and  no  one  could  tell  me."  [Many  agree] 

"Six  months  before,  'you  were  doing  a  great  job.'  Why  didn't  you  tell  me  there  was  a  problem?" 

Employees 

"It  is  worthless.  I  talk  to  my  boss  every  day  about  what  she  wants,  so  I  constantly  get  feedback. 

Unless  I'm  getting  feedback  from  the  panel  at  the  interim,  it's  worthless." 

Supervisor 
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"The  appraisal  process  is  the  most  important  thing  NSPS  has  given  us.  It  has  improved  communication 

and  effectiveness.  It  has  forced  communication." 

Commander 

Self-Assessments 

There  were  mixed  responses  on  the  value  of  the  self-assessment.  Some  employees  expressed 
appreciation  for  it  as  a  means  of  showing  the  supervisors  their  work  progress/scope  and  as  a 
way  to  become  more  involved  in  one's  contribution  to  the  mission.  Other  employees  did  not 
see  the  value  of  the  self-assessment,  stating  that  it  does  not  reflect  one's  true  job  and  that  the 
self-assessment  is  only  for  the  pay  pool  panel  as  a  means  to  determine  rating  and  payout.  Some 
employees  expressed  the  belief  that  the  supervisor  was  responsible  for  documenting  employee 
contributions,  not  the  employee.  Some  commanders  believed  the  appraisal  process  is  NSPS' 
most  valuable  tool.  Supervisors  generally  agreed  that  the  self-assessments  are  valuable. 

"It  takes  a  lot  of  time  to  do  the  self-assessment.  I  am  not  sure  what  role  it  played  in  determining  my 

rating  and  payout." 

"I  thought  it  was  great....  Sometimes  my  supervisor  wants  the  end  product  and  doesn't  see  how  I  did 

it,  so  the  assessment  provides  that." 

"They  sucked!  I  wrote  up  my  accomplishments  to  reflect  what  I  did  on  my  job .  It  was  like  writing  a 

great  work  of  fiction.  My  direct  supervisor  knows  what  I  do." 

"Whether  I  agree  with  objective  or  not,  having  involvement  with  appraisals  and  provide  feedback 
helped  me  relate,  identify  important  parts  of  my  job,  can  I  know  what  I  need  to  give  to  the  customer. 
It  forced  you  to  answer  those  questions  and  figure  them  out,  within  the  boundaries  of  those 

objectives.  That's  on  the  positive  side." 

Employees 

"First  go-around,  people  had  trouble  articulating  accomplishments.  It  was  useful  for  me  as  a 

supervisor  to  see  where  people  were  having  trouble." 

Supervisor 

Theme  7.  Five  Percent  Cap  on  Reassignment  Pay  Impacts  Incentive  To  Assume  Supervisory 

Positions 

Employees  across  all  sites  declared  that  there  is  little  incentive  to  assume  a  supervisory  position 
for  the  five  percent  pay  increase  limit  on  reassignment  with  a  disproportional  increase  in 
workload.  Employees  also  indicated  that  the  workload  under  NSPS  was  too  great  for  them  to 
consider  taking  on  a  supervisory  role.  Employees  predicted  that  current  supervisors  will  seek 
non-supervisory  positions  for  the  same  pay  without  the  additional  responsibility,  or  will  seek 
non-supervisory  positions  as  technical  employees  in  the  next  higher  pay  band.  Supervisors  cited 
similar  concerns  and  noted  examples  of  supervisors  who  left,  or  are  considering  leaving,  their 
organizations  because  of  the  workload. 

"I'm  less  likely  to  want  to  be  a  team  leader.  There  is  a  huge  difference  in  responsibilities,  but  you  only 

get  a  5%  increase,  and  this  isn't  even  guaranteed. " 

Employee 

"The  big  responsibility  jump  is  at  GS-13  to  GS-14  supervisor.  Used  to  be  a  big  deal.  It's  now  a 
‘ management  reassignment.'  Why  do  that  when  I'm  within  same  pay  band?  And  constrained  salary?" 

"We  have  no  ability  to  compensate  employees  who  are  moving  from  journeymen  to  leadership  in  any 
way  that  is  comparable  to  what  they  were  making  two  years  ago.  I  personally  think  this  is  an 
outrage.  These  are  key  positions.  Being  able  to  attract  leadership  is  critical.  Not  every  position  should 
be  filled  from  outside;  sometimes  internal  employees  are  in  the  best  position  to  fill  these  positions." 

Supervisors 


May  15,  2009 


3-15 


3.  Analysis  of  Data  Collected  During  Site  Visits 


NSPS  Spiral  1  Evaluation  Report 


3.5.2.  Themes  Related  to  KPP  3:  Credible  and  Trusted  System 

This  KPP  addresses  issues  related  to  the: 

■  Transparency  of  the  system 

■  Fairness  of  decisions  and  the  processes  used  to  make  them 

■  Resolution  of  grievances 

■  Handling  of  performance  deficiencies 

■  Compensation  that  ensures  that,  in  the  aggregate,  employees  are  not  disadvantaged 
compared  to  the  GS  system 

Theme  8.  Pay  Pool  Panel  Processes  Lack  Transparency 

Transparency  of  Pay  Pool  Panel  Deliberations 

A  lack  of  transparency  of  pay  pool  panel  procedures  and  a  lack  of  feedback  from  supervisors 
and  the  pay  pool  panel  on  payout  results  hindered  acceptance  of  NSPS  as  a  credible  and  trusted 
system.  A  significant  number  of  employees,  citing  inconsistencies  in  pay  pool  procedures,  were 
concerned  about  unfair  advantage  and  reward.  The  lack  of  transparency  and  understanding  has 
instilled  some  sense  of  mistrust  and  doubt  among  many  employees.  While  many  organizations 
had  good  communications  to  the  workforce,  many  employees  at  other  locations  were  unaware 
of,  partially  aware  of,  or  misinformed  about  pay  pool  panel  processes  such  as  salary/bonus  split 
decisions  and  share  determination.  Some  supervisors  reported  that  neither  they  nor  their 
employees  understood  how  the  pay  pool  panel  process  works  or  why  it  was  used,  while 
supervisors  who  had  sat  on  a  pay  pool  panel  or  employees  and  supervisors  who  had  been 
briefed  by  a  pay  pool  panel  member  were  better  informed.  Employees  by  and  large  wanted  to 
know  how  their  direct  supervisors  rated  their  work  and  were  not  satisfied  by  either  the  lack  of 
explanation  or  the  prohibition  placed 
upon  supervisors  to  not  share  their 
recommended  ratings.  Some 
supervisors  and  employees  saw 
supervisors'  authority  diluted  by  the  pay 
pool  panel  process  while  many 
employees  did  not  understand  how 
supervisors  and  pay  pool  panels 
interacted.  Supervisors  reported  that 
the  amount  of  interaction  between  the 
supervisors  and  the  pay  pool  panels 
varied.  In  some  cases  the  supervisors 
had  very  little  input  beyond  their 
written  assessment  and  received  very 
little  feedback  from  the  panel.  In  other 
cases,  supervisors  were  able  to  meet 
with  the  panel  members.  Pay  Pool 
Panels  and  PRAs  understood  the 
frustration  of  supervisors  whom  the  pay 
pool  panel  were  told  to  reduce  their 
recommended  ratings  but  cited  that 
some  supervisors  had  difficulty 
substantiating  their  write-ups. 

"It  is  unclear  how  the  payout  decisions  are  made  -  how  do  they  decide  what  percentage  goes  to  salary 
and  what  percentage  go  to  bonuses?  This  is  very  important.  There  are  long-term  impacts  for  these 

decisions."  [Much  agreement] 


Issues:  Transparency  of  Pay  Pool  Processes 

(1)  Receiving  little  or  no  information  about  pay  pool,  pay 
pool  processes  or  who  is  on  the  panel 

(2)  Perception  of  a  quota  system  or  forced  distribution  to 
ensure  that  ratings  are  mostly  3s 

(3)  Perception  of  ratings  and  payouts  based  on  the  quality 
of  write-ups  and  not  on  actual  achievements, 
penalizing  employees  with  lower  quality  write-ups 

(4)  Perception  that  non-mission-central  jobs  and  low- 
visibility  jobs  were  likely  to  receive  to  lower  ratings 

(5)  Belief  that  the  pay  pool  compared  the  individual 
performance  assessments  to  other  employee 
assessments,  rather  than  to  established  performance 
indicators  and  benchmarks 

(6)  Lack  of  information  about  process,  consistency,  and 
fairness  of  salary/bonus  split  decision  processes 

(7)  Uninformed  or  uncertain  about  how  employees'  cases 
were  made  before  the  pay  pool  panel 

(8)  Uncertainty  about  the  supervisor's  role  with  the  pay 
pool  panel 

(9)  Concern  that  the  supervisor  be  able  and  willing  to  make 

employees'  cases  fairly  before  the  pay  pool  panel 
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"I  think  the  outline  of  how  they  proceeded  was  put  out,  but  how  they  decided  who  got  how  many 

shares  is  a  mystery." 

"I'm  measured  against  what  people  in  other  departments  did." 

"(The)  panel  is  not  stupid,  but  it's  hard  for  them  to  know  all  the  details  of  all  the  jobs  they  must 

represent. " 

Employees 

"As  a  supervisor,  to  me  it  was  a  black  box." 

"In  the  pay  pool  panel,  when  we  go  up,  we  have  the  impression  we're  competing  with  doctors, 
engineers.  We  have  the  impression  the  base  will  say,  'We  don't  want  to  lose  this  doctor,  so  let's  give 
him  more  money.'  The  biggest  thing  I  saw,  we're  using  the  system  that  would  be  used  by  a  for-profit 
organization.  We're  not.  The  mission  is  the  profit  for  our  organization.  I'm  still  fighting  that.  How  do  I 

do  that  and  still  do  the  mission?" 

Supervisors 

"I  don't  think  employees  have  a  good  grasp  of  what  goes  on  in  the  pay  pool  panel.  There  is  a  lot  of 
thought  that  goes  in  to  each  rating.  Employees  don't  have  full  appreciation  of  what  goes  into  each 

rating." 

"A  key  positive  was  that  communications  were  very  good  on  business  rules.  They  came  out  and 

briefed  business  rules.  Everyone  had  opportunity  to  ask  questions." 

Pay  Pool  Panel  Members 

Beliefs  about  Rating  Determination  Process 

Employees  formed  their  own  perceptions  about  pay  pool  panel  processes,  which  included 
underlying  themes  of  writing  quality,  forced  rating  distribution,  job  importance  and  visibility, 
lack  of  supervisor  influence  in  the  pay  pool  process,  and  representativeness  and  qualifications 
of  pay  pool  panel  members.  Ultimately,  employees  were  concerned  about  the  impact  of  what 
they  believe  may  be  uninformed  decisions  on  career  and  pay.  Pay  pool  panels  reported  that 
using  the  standard  performance  indicators  and  benchmarks  led  to  ratings  that  fell  into  a  bell 
curve. 

"There  needs  to  be  some  feedback  system  in  place  so  that  people  don't  think  they  were  just  being  fit 

into  the  bell  curve.  There  is  a  real  credibility  issue  here." 

Employee 

"We  were  told  everyone  will  be  a  3.  That's  wrong.  They've  already  got  a  preconceived  curve.  Now  all 
your  people  are  not  going  to  fall  in  the  middle;  you  will  get  a  distribution.  The  pay  pool  panel  doesn't 

know  what  you  do,  only  based  on  what  is  written.  And  that's  wrong." 

"I  think  the  pool  is  under  some  pressure  to  get  to  3s.  I'm  sure  you  heard  that  these  guys  were  always 
9s  (getting  the  maximum  rating  under  the  old  system),  now  they're  3s.  There's  got  to  be  more 
marketing  that  this  is  a  huge  transformation  going  on;  I  talked  to  a  pay  pool  panel  member  about 
what  I  need  to  do  to  get  (a)  better  (rating  for  my  employee);  he  said  if  the  guy's  doing  great  he  starts 

at  a  three ...  there's  a  gravitation  to  three." 

Supervisors 

"We  were  able  to  be  fair  and  equitable.  We  were  not  trying  to  meet  a  bell  curve, 
but  as  we  went  along,  it  became  apparent.  There  was  a  shared  expectation 

going  in  that  most  employees  should  be  a  3." 

Pay  Pool  Panel  Member 

"One  of  our  rules  was  to  leave  your  personal  knowledge  out  of  it...  that  makes  it  subjective,  because 

it's  writing  skill  based." 

Implementation  Team  Member 
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Varying  Amounts  of  Rating  and  Payout  Feedback 

Employees  cited  varying  levels  of  feedback  and  explanations  of  ratings  from  supervisors  and 
pay  pool  panels.  Pay  pool  panels  varied  in  providing  information  to  the  workforce  about  ratings: 
some  organizations'  senior  leadership  presented  "road  shows"  with  full  Q&A  sessions,  while 
other  organizations  provided  no  feedback  at 
all. 

"We  put  a  lot  of  effort  in  this.  But  our  boss  was  not 
allowed  to  tell  us  why  our  rating  was  changed. 

They  weren't  told  why.  They  weren't  allowed  to 

"When  we  got  our  payout,  we  received  nothing  from  the  panel.  We  did  not  get  told  why.  No  feedback. 
Feedback  about  how  well  you're  doing,  but  not  about  precisely  what  you  did  or  how  to  do  better. " 

Employees 

"We  did  not  send  out  individual  pay  pool  results  to  employees  (mock  or  live).  To  be  honest  we  just 
didn't  think  about  doing  this.  I  am  not  sure  what  the  impact  would  be.  People  may  look  to  job  hop  if 

we  did." 
PRA 


"We  never  hear  anything  or  get  any  information.  "It’s 
done'  is  all  we  get.  This  leads  to  lots  of  rumors  and  ideas 
about  what  is  being  done  behind  the  curtain. 77 

—Employee,  on  Pay  Pool  Panel  Results  Feedback 

tell  us.  The  word  transparency  has  taken  us  in  the 

opposite  direction." 


Theme  9.  Differences  across  Pay  Pools  Affect  Perceptions  of  Fairness 

A  significant  number  of  employees  across  all  sites,  citing  perceived  inconsistencies  in  pay  pool 
outcomes,  were  concerned  about  unfair  advantage  and  rewards.  Pay  pool  panels,  meanwhile, 
describe  the  actions  they  took  to  ensure  as  consistent  a  process  as  possible.  The  discussion 
below  outlines  respective  perspectives. 

Employee  Perceptions 

Employee  concerns  about  perceived  inconsistencies  in  pay  pool  panel  outcomes  centered 
around:  (1)  policies  and  procedures,  including  rounding  rules,  (2)  different  share  values,  and  (3) 
inconsistent  ratings  and  payout  distributions  across  organizational  pay  pools.  Additionally, 
some  employees  stated  that  local  command  and  pay  pool  policies  constrained  accurate  ratings 
that  benefited  some  but  not  others. 

"All  organizations  are  doing  it  differently.  Different  criteria  for  the  payout  process  and  the  input  and 

who  has  the  say  on  what.  We  should  all  do  it  the  same." 

"It  was  a  surprise  to  see  that  so  many  people  came  out  with  5s,  because  we  thought  you  really  had  to 

walk  on  water  to  get  a  5. " 

Employees 

Supervisor  Experiences 

Supervisors  exhibited  a  greater  understanding  of  the  pay  pool  panel  process.  Some  supervisors 
had  the  opportunity  to  sit  a  sub-pay  pool  panel.  These  describe  some  difficulties  they 
encountered  and  unanticipated  outcomes  as  a  result  of  pay  pool  panel  deliberations.  These 
outcomes  center  on  the  difficulty  with  rewarding  high-performing  individuals. 

"I  was  on  a  supervisor  pay  pool  and  we  only  looked  at  the  high  and  low  ratings.  There  was  a 
possibility  that  some  of  the  3s  should  have  been  4s  compared  to  the  rest  of  the  pay  pool.  No  check 

and  balance." 

"This  is  the  first  time  that  I  did  not  feel  good  about  the  results  of  the  pay  pool  panel  process.  I  felt  like  I 
did  a  good  job  of  reviewing  the  packets  that  came  to  me.  The  top  performer  was  dropped  down  ... 
when  it  goes  to  this  entity  (the  pay  pool  panel)  that  doesn't  know  the  job  that's  what  you  see.  The  star 

performers  didn't  get  what  I  thought  they  should." 
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"It  is  easier  for  some  positions  to  get  higher  ratings  than  others,  especially  HR  Assistants.  Journeymen 
were  hardest  to  rate  because  expectations  were  very  high  for  this  level;  (they)  didn't  always  do 
exactly  what  was  stated  in  the  performance  plans.  Expectations  may  have  been  a  little  too  high  for 

these  levels." 

Supervisors 

Pay  Pool  Panel  Procedures  and  Activities 

Pay  pool  panel  members  identified 
development  of  business  rules  and 
procedures  to  ensure  consistency 
across  pay  pools  as  priorities.  Pay 
pool  panel  members  across  all  sites 
and  organizations  explained  that 
pay  pool  business  rules  were 
developed  based  on  higher-level 
guidance,  were  adapted  from  those 
used  by  other  organizations,  or  were 
changed  as  a  result  of  the  mock  pay 
pool  process.  Several  organizations 
changed  their  process  as  lessons 
learned  emerged.  Performance  review  authorities  were  concerned  with  the  pay  pool  panel 
system's  being  fair,  transparent,  and  equitable.  PRAs  cited  having  a  standardized  process, 
publishing  business  rules  early  on,  engaging  in  constant  consultation  and  collaboration,  and 
having  the  right  people  on  the  pay  pool  panel  as  methods  they  used  to  facilitate  a  fair  and 
equitable  process.  PRAs  were  satisfied  with  the  pay  pool  panel  process,  indicating  that  it  was 
fair  and  equitable,  controlled  for  rating  inflation,  and  resulted  in  on-time  payouts. 

"I  was  pay  pool  manager  for  four  pay  pools.  Most  of  the  discussions  centered  on  what  goes  beyond 
level  3  -  quantity,  quality,  timeliness  -  goes  beyond  what  is  expected.  This  is  a  judgment  call." 

Pay  Pool  Panel  Member 

"Another  big  interest  was  ensuring  that  the  system  was  fair  and  transparent.  People  are  okay  being  a 
3  as  long  as  they  know  it  is  consistent  across  the  board.  One  pay  pool  should  not  have  all  5s  and  be 
getting  a  larger  portion  of  the  pay.  People  need  to  feel  comfortable  with  their  ratings. " 

"The  pay  pool  process  is  fair.  There  are  enough  checks  and  balances.  We  went  with  the  sub-pay  pool 
process  in  the  branches.  This  caused  some  friction  because  decisions  were  overruled.  We  need  to  put 

the  onus  on  the  supervisor  to  get  the  ratings  right." 

Commanders 

"During  the  mock  pay  pools,  pay  pool  managers  felt  the  weight  of  NSPS  was  on  them.  In  the  long  run, 
I  felt  the  work  was  justified  by  the  pay  pool  process.  Giving  a  lot  of  time  to  this  was  the  right  thing  to 
do  because  the  outcome  was  that  employees  felt  they  were  equitably  treated.  We  treated  it  like  a 

promotion  board  where  every  record  was  thoroughly  discussed." 

PRA 

Pay  Pool  Panel  Difficulties 

With  procedures  in  place  in  most  pay  pool  panels,  a  number  of  pay  pool  panels  during  their 
early  pay  pools  still  found  difficulty  with:  (1)  maintaining  consistency  in  the  review  process,  (2) 
comparing  objectives  from  different  organizations  or  jobs,  and  (3)  insufficient  narratives. 

"There  was  continued  frustration.  We  have  a  good  crop  of  civil  service.  Since  we  have 
a  good  crop,  raters  wanted  their  employees  to  fare  well  and  were  passionate  about 
taking  care  of  their  people.  The  ones  that  wanted  to  talk  were  frustrated 
that  their  employees  had  not  garnered  what  they  had  submitted  them  for. 

They  felt  bad  that  they  didn't  carry  the  day  for  their  employee. 


Pay  Pool  Panel  Members'  Perceptions  of  the  Pay  Pool  Process 

Positives 

(1)  Each  panel  had  flexibility  to  manage  the  process 

(2)  The  pay  pool  panel  was  accessible  to  supervisors 

(3)  Panels  strived  to  work  out  any  inconsistencies 
Neutrals 

(1)  Panel  members  gained  an  increased  understanding  of 
the  process  over  time 
Negatives 

(1)  The  panels  did  not  meet  expectations  for  time 
required  in  first  cycle  in  spite  of  planning 

(2)  Too  many  back  and  forths  in  the  review  process;  not 
everyone  involved  in  the  process  knew  what  to  do 
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There  was  lots  of  interest  in  help  to  write  better  objectives." 

Pay  Pool  Panel  Member 

Supervisor-Pay  Pool  Panel  Relationship 

An  indirect  contributor  to  perceptions  of  inconsistent  and  possibly  unfair  ratings  by  the  pay 
pool  panel  deals  with  the  knowledge  employees  have  about  the  interaction  the  supervisor  has 
with  the  pay  pool  panel.  For  a  substantial  number  of  employees,  the  supervisor  has  been  the 
key  role  player  in  their  annual  evaluations.  During  these  early  days  of  NSPS,  many  employees 
did  not  understand  the  role  their  supervisor  played  during  the  pay  pool  panel  review  process, 
assuming  it  was  nominal  or,  worse,  non-existent  beyond  submitting  a  recommended  rating.  In 
many  cases,  this  led  to  employee  perceptions  of  unfairness  in  the  rating  process.  Additionally, 
the  supervisor,  under  policy  restriction  to  not  discuss  if  there  were  pay  pool  panel  changes  to 
the  rating,  seem  hesitant  to  provide  any  explanation  of  rating  to  the  employee  at  all.  The  pay 
pool  panel,  during  its  deliberations,  found  varying  degrees  of  engagement  from  supervisors  in 
responding  to  their  requests  for  substantiation  of  recommended  rating.  Most  supervisors 
indicated  they  knew  that  benchmarks  and  performance  indicators  were  the  criteria  by  which 
ratings  were  determined;  however,  some  supervisors  felt  that  feedback  from  the  pay  pool 
panel  was  not  illuminating.  Lastly,  a  number  of  supervisors  were  resentful  of  the  pay  pool  panel 
process.  They  indicated  they  felt  they  had  lost  their  authority  over  their  employees  as  they  had 
to  justify  ratings  to  a  body  of  assessors,  as  well  as  accept  this  body's  final  judgment  by  signing 
their  own  name  to  the  changes  made. 

"How  much  input  does  our  supervisor  have  during  the  process  to  justify  the  original  rating?  Given 
different  pieces  of  paper  that  have  nothing  to  do  with  each  other ...  it's  all  very  unclear. " 

"The  panel  is  only  looking  at  what  is  on  our  paper.  They  need  to  speak  to  our  supervisors. " 

"The  personality  issue ...  my  boss  doesn't  like  to  hear  'no'  so  he  gives  me  a  bad  evaluation  even 
though  I'm  doing  my  job.  How  do  I  know  that  the  board  gets  to  hear  my  side?" 

Employees 

"I  feel  like  I  lost  a  lot  of  credibility  because  I  told  employees  what  they  could  do  to  get  a  higher  rating 

[during  the  interim  review]  and  then  they  didn't." 

"I  got  clear  guidance  from  the  pay  pool  on  what  the  problem  with  my  ratings  was  and  I  was  able  to  fix 

it  in  one  try." 

"Under  NSPS,  I  lost  my  ability  to  actually  rate  my  people  -  the  board  (pay  pool  panel)  overrides  my 

input." 

"We  provide  the  final  feedback  (to  the  employees)  about  what  you  got  but  I  don't  know  why  (they 
receive  the  ratings  that  they  do).  That's  the  difficult  part  -  we  don't  know  the  basis  of  the  ratings. 
They  (the  pay  pool  panel)  don't  need  to  tell  us  their  concerns  with  specific  individuals,  but  (they  should 

tell  us)  their  concerns  overall." 

Supervisors 

"(The  Pay  Pool  Panel)  would  call  the  raters  to  get  more  information.  Some  people  blew  it  off  and 
didn't  take  the  time  to  fight  for  their  employees.  Others  wanted  to  know  how  to  change  the  write-up 

to  support  the  rating....  A  third  and  very  vocal  group  was  hostile." 

"I  understand  the  frustration  of  signing  something  that  wasn't  your  assessment.  It  is  easy  to  say  that 
they  should  have  developed  better  objectives  or  write-ups.  But  in  some  cases,  there  were  some 
superstar  write-ups,  but  they  did  not  garner  their  recommendation.  Raters  could  understand  that 
there  was  a  system,  that  they  may  not  fare  with  their  original  input,  and  that  they  will  not  get  the 
compensation  they  hoped.  They  could  not  accept  that  they  had  to  sign  a  piece  of  paper  that  was  not 

their  assessment  of  the  individual." 

Pay  Pool  Panel  Members 
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Theme  10.  Employees  and  Senior  Managers  Differ  in  their  Interpretations  of  the  Low  Number  of 

Reconsideration  Requests 

Employees  and  supervisors  had  mixed  knowledge  of  the  reconsideration  process  and  mixed 
responses  about  the  feasibility  and  effectiveness  of  the  reconsideration  process.  Employees' 
knowledge  and  utilization  of  the  reconsideration  process  varied  across  organizations  and  sites. 
Some  supervisors  agreed  that  employees  did  not  understand  the  reconsideration  process.  A 
number  of  employees  indicated  awareness  of  a  process  but  did  not  investigate  it.  A  minority  of 
employees  feared  retribution  if  they  sought  reconsideration.  A  small  minority  of  employees 
indicated  the  lack  of  personal  interface  caused  them  to  drop  their  reconsideration  request. 
Some  employees  saw  a  conflict  of  interest  because,  in  their  understanding,  the  pay  pool 
manager  was  the  reconsideration  authority;  they  saw  no  purpose  in  seeking  reconsideration  in 
this  circumstance.  Employees  across  several  sites  noted  an  inopportune  timing  of  the  limited 
reconsideration  window,  which  fell  during  the  winter  holidays  and  before  publication  of  results 
that  provided  perspective  on  their  scores.  Senior  leaders  interpreted  the  lack  of  questions 
about  ratings  and  payouts  to  indicate  employee  satisfaction.  Employees  reported  that  town 
halls  or  formal  presentations  were  ineffective  venues  for  answering  their  questions. 

"We  got  an  email  when  everyone  was  on  Christmas  holiday  saying  you  have  a  week  to  challenge  the 
rating.  This  was  also  done  before  they  gave  a  summary  of  results.  The  timing  just  didn't  make  sense. " 

“We  were  told  we  had  10  days.  That  was  made  very  clear  for  us. " 

"One  of  my  supervisors  suggested  I  appeal.  I  just  felt  it  was  too  soon  to  even  try  that  issue.  It's  a  brand 

new  system.  I  didn't  feel  it  was  the  right  time. " 

Employees 

"The  thing  that  surprised  me  the  most  was  that  I  expected  a  lot  of  reconsiderations.  We  did  not  get 

that.  I  was  amazed." 

"(The  low  reconsideration  numbers)  mean  people  felt  as  though  they  were  treated  fairly." 

PRAs 

Effectiveness  of  Reconsideration  Process 

Some  pay  pool  panel  members  thought  the  reconsideration  process  was  effective,  but  others 
were  dissatisfied  (e.g.,  lack  of  policy/guidance,  lack  of  independent  fact-finding,  not  enough 
time  to  conduct).  PRAs  judged  the  success  of  the  appraisals  and  payouts  by  the  small  number 
of  reconsiderations.  Some  PRAs  cited  a  lack  of  guidance  about  how  to  implement  the 
reconsideration  process. 

"We  winged  this  (reconsiderations)  on  the  fly.  We  didn't  have  prescribed  best  practices  on  how  to 
handle  reconsiderations.  As  the  pay  pool  manager,  I  was  the  judge,  jury,  and  executioner;  I  didn't 
have  a  support  system.  I  had  the  job  of  the  rating  official,  pay  pool  manager,  and  HR.  We  need  to 

think  about  how  to  get  independence  in  the  process." 

"I  think  it  (the  reconsideration  process)  worked  fine.  One  lesson  learned  was  that  I  processed  the  first 
one  manually.  After  that,  I  met  with  every  individual  to  talk  about  their  case....  I  met  with  some  of 
them  twice.  I  tried  to  keep  it  open  and  let  them  plead  their  case.  Now  we  have  more  time.  Before  we 
had  only  48  hours.  To  be  fair  to  the  employee,  we  can  go  back  and  get  input  from  necessary  people.  I 

think  people  were  getting  a  lot  of  opportunities  to  plead  their  case." 

Pay  Pool  Panel  Members 

"We  had  no  guidance  on  how  to  handle  reconsiderations  for  either  the  pay  pool  or  the  PR  A.  We  would 

like  more  guidance  on  this." 

PRA 
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3.5.3.  Themes  Related  to  KPP  5:  Supporting  Infrastructure 
This  KPP  relates  to  issues  concerning: 

■  Whether  employees  and  managers  are  knowledgeable  about  NSPS  as  a  result  of  training 

■  Whether  the  supporting  infrastructure  provides  common  functionality 

Theme  11.  Difficulty  in  Writing  Performance  Objectives  and  Assessing  Performance 

Difficulty  Writing  SMART  Objectives 

A  prominent  concern  and  frustration  among  employees,  supervisors,  pay  pool  panel  members 
and  implementation  team  members  was  the  difficulty  of  writing  measurable  performance 
objectives.  Employees  were  equally  frustrated  by  the  difficulty  in  exceeding  measurable  and 
non-measurable  performance  objectives. 

"You  make  your  objectives  broad  to  cover  anything  that  could  come  up,  but  that  causes  problems 
when  it  gets  to  the  pay  pool  because  then  they  are  not  specific  enough,  and  it's  hard  to  measure 

progress. " 

"There  are  a  lot  of  constraints  when  it  comes  to  meeting  your  objectives.  Some  things  happen  once  a 

year  and  there  really  is  no  way  to  'exceed.'  You  either  do  it  or  you  don't. " 

Employees 

"(NSPS  would  be)  very  easy  if  you  supervise  an  employee  who  makes  widgets.  In  our  job,  99%  of  what 
we  do  involves  federal  laws.  So  when  100%  compliance  is  minimum,  our  biggest  challenge  is 
convincing  people  who  don't  work  with  us  every  day  that  we're  meeting  them  (our  objectives)." 

"Need  to  have  guidelines  for  what  it  takes  to  exceed  expectations  before  appraisals  are  due." 

Supervisors 

"If  I  am  at  a  low  level  doing  a  routine  task,  I  can  write  an  objective  But  at  a  high  level,  like  most  of 
the  people  (in  our  organization),  it's  a  very  tough  process  because  there  is  no  end  or  measurable  result 

and/or  product. " 

"One  problem  is  that  we  got  a  lot  of  write-ups  with  language  simply  from  the  performance  indicators 
with  no  evidence  to  support  the  rating.  I'm  worried  about  this  for  the  next  year.  People  are  learning 

that  they  can  just  spit  back  this  language.  They  need  to  justify  the  ratings." 

Pay  Pool  Panels 

"Writing  SMART  objectives  was  the  biggest  challenge  (that)  yielded  lots  of  reconsiderations." 

"People  were  inventing  stuff  to  measure  just  so  that  they  could  have  a  (measurable)  objective.  They 

weren't  necessarily  important  things,  but  they  were  measurable." 

Implementation  Team  Members 

Addressing  Entire  Scope  of  Responsibilities,  Emergent  Tasks,  and  Additional  Duties 

Many  employees  were  uncertain  about  how  to  address  emergent  and  non-mission-central 
tasks,  additional  duties,  or  the  full  scope  of  their  primary  responsibilities  with  a  limited  number 
of  performance  objectives.  Supervisors  struggled  with  how 
specific  or  broad  objectives  must  be,  a  point  agreed  upon  by 
pay  pool  panel  members.  PRAs  agreed  that  a  dynamic 
environment  makes  writing  objectives  challenging,  but  that 
specificity  is  desired.  Pay  pool  panel  members  were 
concerned  about  objectives  that  are  appropriate  to  the  employee's  position  in  the  pay  band, 
and  they  cited  issues  regarding  failure  to  substantiate  rating  recommendations  in  the 
assessments.  Employees  across  all  sites  shared  significant  frustration  over  job  outcomes  that 
are  beyond  their  control,  but  for  which  they  are  held  accountable  in  rating.  Employees  had 
mixed  views  on  the  feasibility  and  practicability  of  aligning  performance  objectives  to 
organizational  strategic  goals. 


"Hold  me  accountable  for  what  I  can 
control." 

—Employee,  on  performance 
objectives 
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"It  doesn't  cover  everything  that  you  need  to  do  in  your  job  because  so  much  is  emergent.  Things 
come  up  that  you  have  to  do  and  you  can't  just  not  do  them  because  they  are  not  part  of  your 

objectives." 

"Real  life  interferes  with  meeting  the  objectives.  I  should  be  accountable  if  it  (meeting  an  objective)  is 
under  my  control,  but  not  if  it  is  not ....  (There's)  too  much  emphasis  on  proving  (that  you  met  your 
objectives)  that  we  are  drowning  under  rules.  Hold  me  accountable  for  what  I  can  control. " 

Employees 

"When  we  had  work  plans,  they  had 
objectives,  or  things  we  wanted  them 
(employees)  to  do.  The  contributing 
factors  are  supposed  to  do  that. 

We're  rating  them  on  performance 
objectives,  but  I  think  we're  missing  a 
large  part  of  what  the  employee  does 
outside  the  objective....  That's 
missing  in  NS  PS." 

"I  have  people  doing  very  different  jobs.  How  do  I  compare  them  on  the  same  objectives?  So  I  make 
them  really  general  to  cover  the  work,  but  that  does  not  covey  to  them  what  I  expect  and  want  them 

to  do." 

"My  boss  (a  military  supervisor)  changes  every  year  in  June  or  July.  You  don't  get  to  know  who  your 

boss  is  or  what  their  expectations  are." 

Supervisors 

"Raters  are  not  used  to  having  to  do  so  much  work.  NSPS  requires  you  to  have  to  really  think  about 
setting  goals  for  individual  employees.  There  really  is  a  strategy  and  a  science  of  how  you  write 

objectives." 

Pay  Pool  Panel  Member 

"Our  organization  is  a  dynamic  and  rapidly  changing  organization.  This  makes  it  very  difficult  to  set 

objectives." 

"We  are  rating  one  individual,  but  if  your  program  is  not  doing  well,  it  will  come  out  in  the  ratings  of 

the  individuals." 

PRAs 

Addressing  Achievements  Unrelated  to  Objectives 

A  significant  number  of  employees  had  difficulty  understanding  how  achievements,  such  as 
awards  or  education,  that  are  valuable  but  not  tied  to  objectives  would  factor  into  their 
performance  ratings.  Employees  at  all  sites  were  uncertain  about  the  "appropriate"  number  of 
objectives. 

"I  was  selected  as  civilian  of  the  year,  but  I  was  told  I  was  a  3.  I  got  opposite  signals." 

Employee 

Performance  Near  the  End  of  the  Rating  Period 

A  number  of  employees  and  supervisors  raised  the  concern  that  performance  objectives 
achieved  within  90  days  of  the  end  of  the  rating  period  were  not  being  considered  by  the  pay 
pool  panel  as  part  of  one's  achievements  during  the  rating  cycle.  A  small  number  of  employees 
shared  that  they  had  no  knowledge  that  they  could  change  their  objectives  during  the  rating 
period,  and  thereby  get  credit  for  changing  priorities.  As  well,  a  small  number  of  employees 
confessed  that  they  had  not  fully  capitalized  on  their  ability  to  change  job  objectives  as  needed. 

"It's  self  discipline  on  our  part  because  you  can  change  your  objectives.  But  then  you  have  to  decide  if 

it's  (changing  your  objectives)  really  worth  the  time." 


Employee  1:  "How  do  I  write  taskings  against  my  objectives?  Four  months  on 
special  duty  doesn't  apply  to  my  objectives.  [It]  has  nothing  to  do  to  support 
the  warfighter.  I  don't  feel  that  I  can  say  no  to  a  senior  ranking  official  when  I 
get  tasked  like  this.  That  will  kill  me." 

Employee  2:  "But  your  objectives  can  change." 

Employee  1:  "Can  I  write,  'I  planned  a  retirement  dinner  for  my  SES'?  The 
pay  pool  panel  will  laugh." 

— Employees ,  discussing  emergent  tasks  and  performance  objectives 
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"We  could  not  change  our  objectives  during  the  year  last  year." 

Employees 


Choosing  the  Number  of  Objectives  and  Using  Mandatory  Objectives 

Employees  reported  that  organizations  provided  much,  some,  or  sometimes  no  guidance 
before  and  after  the  first  payout  on  the  "appropriate"  number  of  performance  objectives.  Both 
employees  and  supervisors  raised  issue  about  mandatory  objectives,  but  supervisors  were 
especially  vocal,  citing  that  these  component-,  command-  or  organization-directed  objectives 
were  not  written  to  SMART  standards  and  as  such  were  difficult,  if  not  impossible,  to  exceed. 


"There  was  confusion  about  how  many  objectives  to  include." 


"I  was  given  three  objectives  and  had  no  say  in  them  at  all." 

Employees 

"The  supervisor  (objective)  is  very  hard  to  measure.  It's  hard  to  excel  as  a  supervisor.  Almost  anything 

you  do  is  expected  of  you.  It's  a  3." 

Supervisor 

Aligning  Performance  Objectives  to  Strategic  Goals 

Supervisors  and  some  employees  generally  liked  the  idea  of  aligning  performance  objectives  to 
strategic  goals. 

"You  tie  your  tasks  back  to  your  strategic  plan.  That's  good.  A  year  ago  maybe  five%  were  aware; 

now,  a  lot  more  people  are  (aware  of  it). " 

"Structurally,  going  through  discipline  and  effort,  there's  more  of  an  effort  to  correlate.  Conceptually, 
there's  more  of  a  correlation;  here's  the  objectives,  here's  the  strategy.  There's  structure  there.  But 

the  reality  of  it..." 

Employees 

Use  of  Contributing  Factors 

Employees  asserted  that  guidance  on  contributing 
factors  is  non-existent,  changing,  or  not  followed,  or 
that  they  did  not  know  which  specific  contributing 
factors  to  use  to  enhance  their  performance  plan. 

Some  employees  reported  they  confused 
contributing  factors  with  the  performance  objective 
itself.  Some  supervisors  expressed  liking  contributing 
factors,  but  also  added  that  they  do  not  know  how  to 
document  them  independently  of  the  performance 
objectives. 

"First  they  told  us  no  more  than  3,  then  they  said  use  only  1,  and  then  they  said  to  never  use  the 

technical  (contributing)  factor. " 

Some  employees  were  told  that  they  have  to  meet  all  contributing  factors,  and  that's  not  fair  because 

it's  harder  to  meet  the  objective. 

Employees 

"I  got  six  opinions  across  the  (service  component)  about  how  to  use  contributing  factors.  Why  not  just 
be  a  4?  Even  our  pay  pool  panel,  with  the  same  training,  had  disagreement  on  how  to  use  them." 

Supervisor 

"Contributing  factors  were  very  hard.  You  were  better  off  giving  someone  a  higher  rating  versus 

giving  a  plus  1.  They  were  hard  to  justify." 

Pay  Pool  Panel  Member 


Issues  with  Contributing  Factors 

(1)  Difficulty  understanding  which  and  how 
many  to  use  with  performance 
objectives 

(2)  Difficulty  understanding  the  criteria  for 
+/- 1 

(3)  Lack  of  or  unclear  guidance  on  the  use 
of  contributing  factors  during  the  first 
cycle 

(4)  Changing  guidance  in  the  second  cycle 
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"I  like  the  concept  (of  contributing  factors)  —  that  it  is  a  way  to  capture  over  and  above  what  they  do 
on  the  job.  Also,  we  did  use  it  negatively....  We  like  the  concept,  but  are  struggling  with  how  to  do  it 

properly." 

Pay  Pool  Panel  Member 

Weighting  Objectives 

Organizations  generally  did  not  use  weights  or  used  them  inconsistently.  Some  supervisors  used 
weighting  to  minimize  potential  negative  impact  of  non-measurable  mandatory  objectives. 
Some  members  of  implementation  teams  saw  weighting  as  a  way  to  bring  a  more  well-rounded 
focus  on  the  mission  as  well  as  primary  job  tasks. 

"/  want  the  flexibility  to  weight,  like,  if  one  of  my  objectives  is  really  important,  it  should  count  for 

more. " 


"You  get  40%  for  the  objectives  you  have,  and  only  40% 

(weighting)  can  be  the  top  amount.  If  more  than  40%  is 
spent  doing  your  job,  then  you  have  to  have  all  these  other 
things.  [Many  nods  and  yeses]  The  weighting  is  what's 
detrimental.  What  if  60%  of  your  job  is  your  job  (one 
objective)?  You  can  only  get  so  much  blood  out  of  a 

stone." [Much  agreement] 

Employees 

"The  mandatory  objectives  are  not  SMART.  That's  why  we  weight  at  10%." 

Supervisor 

"Weighting  of  job  objectives...  supervisors  and  managers  have  tried  to  game  the  system  with  these 
weights,  so  weights  are  now  looked  down  upon.  The  same  can  be  said  for  contributing  factors." 

"(I)  learned  a  big  lesson  on  weighting  of  objectives.  Max  weight  of  any  objective  is  40%;  minimum  is 
10%.  Have  one  primary  objective  for  everyone....  It's  a  great  leveler.  It  gets  everyone  focused  on  what 

they  need  to  be  doing  to  support  the  mission." 

Implementation  Team  Members 

Theme  12.  Workforce  Praises  the  Mock  Pay  Pool  but  Identifies  Training  Weaknesses 

Mock  Pay  Pool 

Pay  pool  panel  members  and  supervisors  who  participated  in  pay  pools  praised  the  mock  pay 
pool  as  valuable  for  teaching  lessons  about  performance  objectives  and  interim  assessments. 
Some  PRAs  stated  they  focused  their  mock  feedback  on  the  processes,  rather  than  providing 
feedback  to  employees.  Employees  and  supervisors  noted  that  the  mock  is  valuable  only  if 
feedback  is  provided. 

"The  mock  pay  pool,  we  got  no  feedback  about  what  our  numbers  would  have  been.  And  we  went 

through  the  whole  process  of  doing  it.  No  feedback  at  all." 

Employee 

"I'm  still  looking  for  training  on  (writing)  job  objectives.  If  we'd  gotten  interim  feedback  from  the 

mock  pay  pool,  that  would  have  gone  a  long  way." 

Supervisor 

"After  mock  pay  pools,  we  let  people  come  and  ask  questions.  It  was  good  for  showing  the 
transparency  of  the  system  and  good  for  us,  too,  to  help  us  articulate  our  process  and  improve  our 

deliberations." 

"In  (directorate  name),  we  worked  with  the  supervisors  to  do  a  better  job  with  their  writing  and 

showed  them  where  they  weren't  on  track  with  their  objectives." 

Pay  Pool  Panel  Members 


Employee  1:  We  worked  with  the  supervisor. 
We  also  had  weights  on  the  objectives  and 
could  challenge  them. 

Employee  2:  We  also  had  them  (in  our 
department). 

Employee  3:  Others  were  not  given  that  choice. 
— Employees ,  discussing  the  use  of  weighting  in 
their  deportments 
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"Mock  pay  pools  were  great.  Probably  biggest  debate  was  whether  to  force  a  bell  curve.  We  really 
needed  to  ensure  the  value  of  the  share  was  sustained;  otherwise,  it  hurts  all  employees." 

Pay  Pool  Panel  Member 

"When  we  did  the  mock,  we  looked  across  pay  pools  and  showed  the  pay  pools  their  results  compared 
to  the  other  pay  pools.  If  one  organization  was  giving  more  5s,  their  share  values  were  lower.  We 
showed  them  the  data  and  this  helped  them  to  understand  the  implications  and  how  it  hurts  their 
people.  We  could  show  them  that  if  we  had  done  things  differently,  their  employees  would  have  done 

better  monetarily." 

PRA 

Performance  Plan 

Employees,  supervisors,  and  implementation  teams  criticized  the  examples  in  performance 
objective  training  as  being  too  "production  oriented"  and  not  relevant  to  their  job  types. 
Supervisors  held  mixed  views  on  "soft  skill"  training.  Implementation  teams  suggested  that 
employees  and  supervisors  be  trained  on  pay  pool  processes  to  increase  understanding. 

"They  tried  to  give  us  a  few  examples,  but  it  was  hard  to  interpret  them  into  our  own  fields.  Not  just 

us,  but  hard  for  our  supervisors,  too." 

Employee 

"Employees  will  tell  you  that  training  doesn't  prepare  them  for  how  to  write  up  performance.  They  do 
everything  from  repeating  job  descriptions,  which  does  nothing,  to  trying  to  pick  one  event  that 

depicts  work  ethic." 

"Training  was  decent  on  writing  SMART  objectives.  Training  was  lacking  on  the  pay  pool  process  and 
the  implication  of  decisions  you  make  on  points  and  shares.  It  wasn't  until  the  end  with  the  CWB  that 
I  started  to  get  an  appreciation  of  the  ramifications  of  decisions.  Some  form  of  training  would  have 

been  helpful  on  this." 

Supervisors 

"Need  better  training  on  objectives.  The  nature  of  our  work  does  not  allow  for  hard  objectives,  and 
you  depend  on  others  to  get  your  job  done.  We  need  more  training  on  this.  The  iSuccess  tool  is  still 

limiting." 

Implementation  Team  Members 

Supervisor  and  Trainer  Effectiveness 

Employees  cited  varying  degrees  of  supervisor  knowledge  and  involvement  in  training, 
assisting,  and  coaching  employees.  Employees  generally  acknowledged  that  supervisors  who 
had  previous  experience  in  NSPS  or  in  a  demo  project  were  very  helpful  for  writing 
performance  plans. 

"/  wrote  my  objectives  and  submitted  them  to  supervisor.  He  looked  at  them,  we  discussed  them,  and 
he  provided  feedback....!  don't  have  a  problem  with  the  objectives  because  I  had  input  on  them. " 

"(There's)  more  rapport  with  supervisors.  Now  they  have  to  pay  attention  to  what  I'm  doing. " 

"Early  on,  we  were  being  trained  by  contractors  who  didn't  know  our  own  terminology.  We've  seen 

steady  improvements  since  then." 

Employees 

"When  I  did  my  employee,  (he  asked)  what  will  it  take  to  do  a  4  or  5?  I  couldn't  answer.  I  felt  I  failed 

as  a  supervisor.  What  will  it  take?  I  don't  know." 

Supervisor 

Training  Weaknesses 

Supervisors  and  employees  identified  inconsistent  trainer  quality  as  a  problem.  In  some 
instances,  trainers  taught  approaches  to  writing  performance  objectives  that  were  inconsistent 
with  their  organization's  approach.  The  workforce  identified  the  Performance  Appraisal 
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Application  as  an  area  where  more  hands-on  training  would  be  beneficial.  Another  area  with 
strong  demand  for  training  was  writing  performance  objectives. 

"I  didn't  feel  like  there  were  connections  between  what  I  got  in  training  and  what  I  was  told  in 

directorate." 

"I  had  an  idea  based  on  the  class  that  I  went  to.  But  then  the  town  hall  meeting  led  me  to  understand 
that  what  I  learned  in  training  was  not  accurate.  But  that  was  a  good  thing.  For  example,  in  class  they 
told  us  that  we  would  be  compared  to  everyone.  But  in  the  town  hall,  they  said  you  were  compared  to 

your  objectives." 

"You  came  in  to  MyBiz  (PAA)  and  (I)  was  helpless  as  it  relates  to  my  specific  organization.  There  were 
no  samples  of  objectives,  but  we  needed  these.  We  also  need  examples  of  how  to  write  appraisals  and 

self  assessments. " 

"( We  received  training  on)  pay  pool  and  how  percentage  is  divided  up.  A  person  came  in  to  explain  it 
to  us.  When  he  broke  it  down  and  where  percentage  goes  and  why,  it  makes  it  a  lot  better  and  doesn't 
seem  so  bad.  I  think  it  would  be  better  if  everybody  really  saw  how  the  formula  works  and  how  they 

figure  out  the  percentages  (what  is  going  to  salary  and  bonus)." 

“We've  always  been  taught  that  5  was  the  highest,  but  then  I  heard  that  some  people  were  given  6 
shares.  If  a  6  is  possible,  I  want  a  6— not  a  5.  I'll  work  towards  anything;  you  just  need  to  tell  me  what 
I'm  working  towards.  I  thought  that  if  you  got  a  rating  of  5,  you  could  only  get  5  shares." 

Employees 

"One  of  the  things  we  need  training  on  is  rewriting  performance  objectives.  For  supervisors  below  me. 
We  as  an  organization  do  not  do  well;  we  do  PDs  (Position  Descriptions)  and  we  don't  assess  on  PDs." 

Supervisor 

"The  training  artificially  set  high  expectations.  We  were  told  that  if  we  wrote  objectives  in  the  SMART 
format,  everyone  could  be  a  role  model.  That's  just  not  true.  Role  model  is  an  exceptional  case." 

Pay  Pool  Panel  Member 

"Every  supervisor  and  employee  needs  pay  pool  training  because  it  gives  perspectives  on  how 

important  the  supervisor's  write  up  and  self-assessment  are." 

Implementation  Team  Members  "The  support  from  contractors  can  be  confusing  because  they  convey 
different  messages  about  doing  certain  things  (e.g.,  how  to  write  objectives)." 

Commanders 

Supplemental  Training 

A  majority  of  organizations  initiated  informal  training  or  networked  with  organizations  already 
in  NSPS  for  guidance,  especially  in  writing  performance  objectives.  Some  organizations 
supplemented  NSPS  training  with  soft  skill  training. 

We  had  an  offsite  with  all  people  and  talked  about  what  a  good  objective  was  and  what  an  objective 

shouldn't  be.  But  this  was  only  one  division/group." 

"I  belong  to  a  small  admin  group.  We  got  together  and  we  formed  a  set  of  common  objectives  that 
each  of  us  was  able  to  take  back  and  tailor  to  our  individual  jobs.  This  was  very  helpful  for  us." 

Employees 

"One  class  that  was  supposed  to  prepare  us  for  NSPS  (basic  soft  skills).  It  was  just  propaganda 

designed  to  get  us  through  anger  and  resignation  phases." 

Supervisor 

"We  communicated  with  supervisors  throughout  the  year.  What  helped  a  little  bit  was  that  as  things 
come  up,  we  would  make  a  connection  to  NSPS  -  'how  would  this  be  covered  by  NSPS?'  We  have  a  lot 
of  NSPS  training  going  on  this  year  to  discuss  whether  an  example  is  something  that  is  going  beyond 
or  what  we  would  expect.  We  took  pieces  of  what  we  do  and  talked  about  how  the  work  ties  back  to 
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NSPS  - ' where  is  the  threshold  between  what  you're  getting  paid  to  do  versus  something  that  would 

carry  weight  in  the  evaluation?'" 

Pay  Pool  Panel  Member 

"When  we  first  started,  we  realized  there  would  be  a  significant  training  requirement.  When 
employees  said  'oh,  there  is  all  this  training,'  I  would  say  how  much  training  did  you  have  on  the  GS 
system?  Probably  not  much.  Since  were  changing  cultures,  we  needed  training,  so  we  developed  an 

integrated  training  plan  involving  all  the  components." 

Implementation  Team  Member 

Lag  Time  and  Training  Effectiveness 

The  time  lag  between  training  and  application  reduced  employees'  ability  to  retain  information 
and  apply  lessons  learned. 

"We  were  being  shuffled  to  training  over  and  over 
six  to  12  months  before  conversion.  The  early 
training  classes  (NSPS  101,  writing  objectives)  really 
didn't  mean  anything  to  me.  It  wasn't  until  we  got 
really  close  to  converting  is  when  my  supervisor  said 
this  is  how  we're  going  to  do  it  and  my  organization 
started  to  develop  ground  rules.  What  really  plugged 

me  in  was  my  organization  saying  what  to  expect  right  before  conversion.  I  would  have  like  to  have 
been  informed  as  early  as  possible  -  not  just  one-two  months  before.  I  probably  learned  more  from 
the  organization  than  the  training.  I  felt  that  the  organization  did  a  very  good  job  as  far  as  informing 

the  employees  and  telling  us  what  to  expect." 

"They  tried  to  tell  us  what  to  expect,  but  it  keeps  changing  (e.g.,  retroactive  changes).  That's 
frustrating.  I  think  everything  is  okay, but  then  they  change  it.  It  was  pretty  good  regarding 
supervisors  sharing  information  with  us,  but  it  is  frustrating  when  everything  changes 

and  they  (the  supervisors)  don't  even  know." 

Employees 

"We  did  it  (training)  before  the  issuances  were  finalized,  so  there  was  a  lot  of  7  don't  know.'...  Those 
doing  the  training  couldn't  answer  half  of  the  questions...  (it  seemed  like  we  were)  doing  change 
management  for  change  management's  sake.  Not  really  going  to  change  the  culture.  That  combined 

with  the  change  itself  created  more  angst." 

Implementation  Team  Member 

"The  training  came  out  in  pieces.  We  need  an  encyclopedia  of  all  the  required  training  moving 
forward.  We  need  automated  and  refresher  modules.  Anything  that  we  can  do  online  is  great." 

Commander 


"I  did  NSPS  101  and  the  training  on  writing 
objectives  and  self-assessments.  When  I  was  in 
the  training,  it  was  good.  I  felt  confident  with  it. 
When  it  came  down  to  implementing  those 
things,  that's  when  it  all  fell  apart." 

— Employee ,  on  training 


3.5.4.  Additional  Themes 

Participants  had  three  categories  of  additional  comments.  These  themes  generally  span  KPPs: 

■  The  amount  of  time  required  by  NSPS 

■  The  automated  tools 

■  OveralleffectsofNSPS 

Theme  13.  Changing  Policies  and  Communications  Processes  Reduced  NSPS  Understanding 

There  were  mixed  views  on  the  effectiveness  of  NSPS  communications  before  and  during 
implementation  and  during  the  rating  and  payout  period. 

Impact  of  Shifting  Policies 

An  issue  across  virtually  all  sites  and  groups  was  the  NDAA  change  to  payout  policy  enacted 
after  initial  spiral  payout  decisions.  Leadership  and  the  workforce  were  united  in  stating  that 
this  policy  shift  caused  confusion  and  reduced  trust  in  NSPS.  Supervisors  and  employees  stated 
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they  were  both  overwhelmed  and  frustrated  by  the  volume  of  information  regarding  NSPS, 
particularly  emails  during  the  initial  implementation  and  other  key  periods. 

"NSPS  could  be  a  great  system  after  a  couple  of  years,  but  the  rest  of  us  are  suffering 

in  the  mean  time  when  the  rules  are  being  figured  out." 

Employee 

"One  of  the  problems  with  training  was  that  training  was  happening  as  policy  was 
being  generated.  When  meaty  questions  (arose)  like  how  pay  pool  works, 
the  answer  was,  'We  haven't  figured  that  out  yet.'  The  answers  came  after  training." 

Supervisor 

"The  change  to  the  Element  1-Element  2  split  is  one  of  the  worst  things  they  could 
have  done  in  terms  of  getting  buy-in....  I  basically  look  at  it  now  like  everything 

is  at  the  will  of  the  DoD  and  we  won't  be  consulted." 

"Initial  flexibility  has  been  eroded  by  the  NDAA  decision  about  Element  2  payout." 

Pay  Pool  Panel  Members 

"We  did  it  (training)  before  the  issuances  were  finalized,  so  there  was  a  lot  of 
'I  don't  know'...  Those  doing  the  training  couldn't  answer  half  of  the  questions... 

(It  seemed  like  we  were)  doing  change  management  for  change 
management's  sake.  Not  really  going  to  change  the  culture.  That,  combined  with  the 

change  itself,  created  more  angst." 
Implementation  Team  Member 

"One  concern  for  the  future  is  the  last-minute  decision  on  the  GPI.  This  could  happen  every  year.  How 
can  we  set  a  target  share  value  with  an  unknown  budget?  We  would  like  a  mechanism  for  setting  the 

target." 

PRA 

Information  Flow 

Information  flow  was  thorough  and  continuous  at  virtually  every  site,  with  some  exceptions  for 
final  payout  information.  Implementation  teams  were  instrumental  in  disseminating  basic  and 
updated  information  to  employees  and  leadership.  PRAs  supported  full  information  flow  to  the 
workforce.  Commanders  generally  worked  to  anticipate  issues  for  employees  and  to  keep 
senior  staff  informed.  Nearly  all  employees  stated  that  information  flow  during  and  after 
implementation  was  overwhelming. 

"It  (information  about  changes  to  NSPS  policy  and  procedure)  was  there.  But  sometimes  there's  an 

information  overload.  Pinpoint  what  you  need  for  us  to  know." 

"Leadership  did  all  they  could  to  try  and  communicate."  [Many  agree.) 

"We’re  pretty  low  informed.  Supervisors  will  eventually  come  around,  but  they're  still  old-school 
supervisors  who  won't  adjust.  Once  new  people  come  in,  it  will  be  a  good  thing." 

Employees 

"We  shared  the  business  rules  with  the  workforce  where  we  could.  We  received  guidance  and  business 
rules  from  the  Executive  Steering  Committee.  We  built  on  consensus  and  published  everything, 
including  statistics  at  the  end  of  the  payouts.  We  briefed  generals  and  the  SES  community  on  the 
results  of  the  payouts  and  how  the  different  bands  were  affected.  There  was  focus  on,  'Are  we  taking 

care  of  YA-ls  and  junior  YA-2s?'  This  was  of  particular  concern." 

Commander 

Communications  To  Ensure  Understanding 

Most  organizations  used  town  hall  meetings  to  brief  the  workforce  on  the  implementation 
process  and  on  rating  and  payout  results.  Town  halls  were  somewhat  effective  in  disseminating 
information,  but  were  not  mandatory  and  left  some  people  poorly  informed.  Road  shows  with 
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senior  leadership  conducting  question-and-answer  sessions  were  effective  in  ensuring 
understanding  of  rating  and  payout  information  and  reducing  any  morale  issues.  The  various 
modes  by  which  information  was  provided  did  not  necessarily  ensure  workforce  understanding. 

"The  briefing  showed  results,  but  it  didn't  provide  any  explanation  to  help  you  understand." 

"They  really  did  a  lot  to  try  and  get  the  message  out  -  it's  a  completely  new  system  that  is  hard  to 

explain." 

Employees 

"One  of  the  things  we  used  initially  was  written  communication.  There  were  also  town  halls  in  each  of 
the  pay  pools.  As  we  matured,  we  did  brown  bags  and  had  NSPS  updates  via  e-mails.  We  had  a 
website.  Things  were  too  fluid  so  we  posted  everything  on  the  website.  We  posted  business  rules.  We 
had  a  link  where  employees  could  ask  questions  and  sometimes  they  would  get  a  personal  call.  We 
had  a  message  for  senior  leaders  and  training  for  them.  Any  time  any  leader  wanted  to  meet  or  had 

an  issue,  we  would  meet  with  them  or  form  a  cross-functional  team." 

"Communication  is  important  -  the  fact  that  we  gave  employees  as  much  information  as  we  had.  In 
some  cases  it  was  frustrating  when  we  didn't  have  the  info  to  give  because  it  was  still  evolving.  We 

won't  have  that  problem  in  the  future." 

Implementation  Team  Members 

"Even  with  all  of  that  communication  (emails,  town  halls,  blogs),  there  was  a  discord  between  the 

information  they  were  given  and  what  they  understood." 

Commander 


Theme  14.  Perceptions  of  IT  Support  Tools 

The  Performance  Appraisal  Application  (PAA),  used  to  support  performance  management 
processes,  experienced  many  technical  problems  during  the  first  and  second  cycles,  impacting 
supervisors'  and  employees'  ability  to  prepare  required  performance  plans,  interim  reviews, 
and  self-assessments.  Many  employees  and  supervisors  cited  frequent  inability  to  access  the 
tool;  an  interface  that  was  not  user  friendly,  including  cumbersome,  non-intuitive  navigation; 
poor  layout  of  screens;  loss  of  data;  lack  of  remote  connectivity  because  of  the  CAC  card 
restriction;  and  one-person-at-a-time  access.  The  PAA's  technical  difficulties  persisted 
throughout  the  second  version's  release,  prompting  some  organizations  to  avoid  full  use  of  the 
PAA  and  instead  using  paper-based  approaches.  Site  visit  findings  indicated  that  PAA  use  was 
further  complicated  by  a  lack  of  hands-on  training  and  outdated,  or  difficult-to-use,  user  guides. 

Other  NSPS  automated  tools  include  the  Readiness  Tool  and  the  Compensation  Workbench 
(CWB).  The  CWB  supports  pay  pool  panel  deliberation  processes.  The  Readiness  Tool  assists  the 
organizational  leadership,  implementation  teams,  and  HR  points  of  contact  in  preparing  for 
implementation,  sharing  information,  and  monitoring  readiness.  The  field  used  both  the 
Readiness  Tool  and  the  CWB  with  few  issues.  Appendix  G  provides  additional  information  on 
other  NSPS  automation. 

The  bulk  of  site  visit  comments  referenced  the  PAA.  MyBiz  is  an  interface  to  the  PAA.  Some 
quotations  below  refer  to  "MyBiz"  as  employees  often  used  these  terms  interchangeably. 

Training 

"No  direct  training  on  the  PAA  given.  No  manual  on  MyBiz  (PAA)  was  offered." 

"The  training  provided  (by)  the  PAA  User  Guide  (177  pages)  is  not  accurate.  It  doesn't  match  the 
application.  We  did  get  some  other  guidance  from  the  personnel  office  this  time  around." 
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Effects  on  Implementation 

"The  PAA.  It  was  un-user  friendly.  It  didn't  destroy  Implementation, 
but  it  wasn't  good  when  trying  to  get  it  out  and  folks  (were)  trying  to  do  performance  (plans)." 

"About  80%  of  the  complaints  I  hear  (are)  to  fix  MyBiz  (PAA).  Make  the  IT  tools  more  user-friendly.  A 

lot  of  people  lose  the  forest  through  the  trees  in  MyBiz  (PAA)." 

Effects  on  Mission 

"My  folks  spend  up  to  two  days  to  get  their  interims  in  the  system." 

"System  is  frantic.  You  are  constantly  chasing  things.  Don't  have  time  to  do  your  real  job  because  you 

are  having  to  chase  down  the  system." 

"There  were  lots  of  questions  about  PAA  and  how  to  use  it....  We  had  two  people  on  phones  almost 

full  time  answering  questions." 

"I  spend  more  time  helping  people  with  the  system  than  actually  rating  them." 

"...  (W)e  are  doing  interim  reviews  right  now,  and  they  are  releasing  upgrades  (to  the  software  - 

making  it  CAC  card  enabled).  Somebody  is  not  thinking." 

Tool  Capacity  and  Accommodation 

"Once  I  have  the  objectives  in  the  system,  I  can't  tweak  them  - 
only  the  rater  can  change.  But  raters  don't  have  time  to  change  all  the  objectives  for  everyone." 

"Another  problem  was  that  the  system  would  go  to  sleep....  Learned  the  hard  way  that  we  need  to 

write  the  objectives  in  Word  and  then  cut  and  paste  into  the  PAA." 

"If  you  have  one  misstep,  it  sets  you  back  3-5  steps." 

"HLRs  (Higher  Level  Reviewer)  cannot  see  at  a  glance  (in  PAA)  the  status  of  all  the  performance  plans 
they  are  supposed  to  review,  so  they  can't  see  who  is  in  compliance  and  who  isn't." 

"The  technical  aspects  drove  me  nuts.  You  are  sending  things  back  and  forth  and  sometimes  it  gets 

lost  in  the  system." 

Theme  15.  NSPS  Performance  Management  System  Demands  Significant  Organizational 

Resources 

Time  and  Administrative  Burden 

The  time  and  administrative  burden  for  NSPS 
implementation  and  administration  was  the 
first  issue  raised  by  virtually  all  sites.  A  majority 
of  employees  and  supervisors  and  many  pay 
pool  panel  and  implementation  team  members 
strongly  felt  that  NSPS  is  extremely  labor  intensive.  The  time  demand  was  complicated  by  staff 
shortages,  travel,  supervisors'  non-supervisory  job  demands,  military  supervisor  rotations,  and 
ops  tempo.  Others  offered  that  time  spent  on  NSPS  requirements  took  away  from  productivity 
and  the  mission  and  impinged  upon  personal  time.  Some  supervisors  and  employees  cited  an 
increased  workload  for  HR  professionals,  especially  in  hiring  within  a  broad  pay-banded  system. 
Some  pay  pool  panel  members  thought  that  the  time  requirements  may  be  unsustainable  long¬ 
term. 

"In  the  GS  system,  my  involvement  in  appraisal  was  about  an  hour  per  year.  My  shop  is  undermanned 
now.  Now,  I  have  to  put  in  40-50  hours  per  year  on  (my)  appraisal  and  my  boss  is  putting  in  40-60 

hours  per  year  on  me.  And  because  of  NSPS,  I  can't  meet  my  objectives." 

"Under  NSPS,  you  have  a  minimum  of  13  pages.  That  is  a  metric  on  how  time-consuming  this  is  for 
everybody  involved.  One  of  the  challenges  is  how  to  master  that  and  keep  up  with  the  high  ops  tempo 

in  headquarters." 


"It  is  an  extreme  distraction  to  supervisors  who  are  also 
action  officers.  The  reality  is  that  we  don't  have  enough 
people  to  handle  the  mission  and  have  dedicated 
supervisors  to  NSPS.  My  heart  breaks  for  leaders  when  I 
see  them  have  to  spend  weeks  to  deal  with  pay  pools." 

— Employee ,  on  time  and  administrative  demands  of  NSPS 
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"I  had  meetings ...  where  I  needed  senior  leadership  representation,  but  they  couldn't  go  because  they 
were  on  call  for  the  pay  pool  panel.  That  resulted  in  (senior  leaders)  not  being  (available),  which 

affects  (the  success  of)  mission." 

Employees 

"I've  spent  more  overtime  on  NSPS  than  any  other  project.  My  worry  is  finding  time  to  really  focus  on 

NSPS." 

"We've  got  to  find  a  way  to  not  let  the  process  become  the  end.  We  have  more  important  jobs  to  do." 

"The  hiring  process  is  very  burdensome.  You  get  a  list  of 200  employees ...  and  the  supervisor  has  to 

wade  through  that.  There's  no  screening  anymore." 

Supervisors 

"Deployments  force  people  (substitutes)  into  the  pay  pool  (panel)  -  not  fair  to  that  person  and  not 
fair  to  employees.  They  are  brand  new  and  they  don't  know  them.  Ops  tempo  of  a  typical  base  and 

the  rotation  of  the  military  leadership.  It  sets  up  for  a  perfect  storm." 

"It  takes  a  lot  of  personal  time.  I  spent  two  weeks  with  the  sub-pay  pool.  This  is  undoable  in  the  long 
term.  It  needs  to  be  more  efficient.  We  are  going  through  each  appraisal  line  by  line  and  analyzing  it." 

Pay  Pool  Panel  Members 

"Pay  for  performance  requires  you  to  spend  time  with  employees,  and  there  are  not  enough  hours  in 

a  day." 

"Had  to  pull  people  off  their  primary  jobs  to  do  NSPS,  and  we  did  not  get  any  additional  resources.  We 

did  not  plan  on  this." 

Implementation  Team  Members 

"I  acknowledge  that  it  takes  time.  I  don't  know  if  that  time's  a  bad  thing.  Pay  pool  training,  mock 

training ...  as  we  become  more  comfortable  I  hope  it  drops  a  little." 

Commander 

"NSPS  was  bigger  than  I  or  anyone  thought.  And  everyone's  plates  are  so  full  right  now.  The  ops 

tempo  is  high.  This  broke  the  camel's  back  for  some  folks." 

PRA 

Return  on  Investment 

Employees  and  supervisors  expressed  doubt  about  a  positive  return  on  investment  for  the  time 
and  energy  needed  to  meet  NSPS  requirements.  With  the  preponderance  of  ratings  around  3, 
employees  and  supervisors  questioned  the  time,  effort,  and  impact  on  mission  as  leadership 
and  the  workforce  work  to  fulfill  performance  plans,  interim  reviews,  self-assessments  and  pay 
pool  panel  requirements.  Commanders  generally  withheld  judgment  about  the  return  on 
investment  but  many  want  more  time  to  let  NSPS  settle  in  and  prove  its  worth. 

"Are  we  meeting/keeping  pace  with  the  GS  schedule?  Probably.  But  there  is  so  much  more  work  here, 

I  am  not  sure  it's  worth  it." 

"As  a  taxpayer  I'm  appalled  about  the  amount  of  time  and  effort  going  into  it  if  it's  (NSPS)  going 
away.  Plus  the  massive  stacks  of  paper  for  the  pay  pool  panel.  Is  there  a  cost  benefit  analysis  (being 

done)  for  having  our  leaders  tied  up  for  weeks  not  available  to  us?" 

Employees 

"(The  process)  takes  10  times  more  time,  without  getting  the 
results.  The  most  common  feedback  from  employees  is  that 
everyone  is  spending  so  much  time  on  it  (the  performance 
management  process)  but  not  getting  the  returns  they  were 

expecting." 


"There  were  one  to  two  people  who  got  a 
5  and  almost  everybody  got  a  3.  That's  a 
lot  of  work  for  everyone  to  be  average." 
—Employee,  on  return  on  investment  in 
NSPS  in  his  organization 
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"It  takes  a  whole  lot  of  time  to  do  assessments.  I  have  second-level  supervisor  responsibilities  with  37 
people.  It  took  me  two  weeks,  and  I  don't  know  if  the  results  were  appreciably  better." 

Supervisors 

"The  time  spent  is  probably  way  out  of  balance  with  the  payback.  The  few  that  are  getting  5s  are 

getting  a  big  chunk.  We  need  to  get  better  at  that." 

Pay  Pool  Panel  Member 

"It  is  extremely  labor  intensive  at  all  levels...in  an  environment  where  you  have  a  lot  of  deployments 
and  PCSs,  it  creates  a  lot  of  turmoil  in  the  process...just  maintaining  the  hierarchy  intact  is  a  huge 

workload  for  the  personnel  folks." 

Implementation  Team  Member 

"The  payback  will  be  worth  it  if  NSPS  will  be  allowed  to  continue.  Some  form  of  NSPS  has  to  stay. 

Don't  give  us  the  GS  system  back." 

"I  am  optimistic  for  NSPS  here.  I  think  this  will  help.  We  are  not  there  yet,  but  I  think  we  will  be.  It  will 
help  to  address  poor  performers.  But  we  have  also  lost  some  high  performers.  We  also  have  problems 

getting  good  supervisors  with  the  5%." 

Commanders 


Theme  16.  Expected  Long-Term  Effects  and  Outcomes 

The  expected  effects  and  outcomes  for  NSPS  vary.  These  are  grouped  by  general  categories  in 
the  section  below. 

Predictors  of  Organizational  Success  and  Climate 

Some  employees  predicted  that  lack  of  clarity  for  career  progression  in  NSPS,  pay  raise 
uncertainty,  the  option  to  move  to  non-NSPS  organizations,  some  supervisors'  lacking  writing 
skills,  and  a  new  and  more  competitive  workplace  climate  could  negatively  impact  retention 
and  job  satisfaction,  thereby  hurting  the  organization.  Commanders  and  PRAs  generally  wished 
to  pursue  NSPS  to  maturity,  yet  they  also  recognized  the  disincentive  for  employees  to  assume 
supervisory  positions,  the  cost  of  turnover  by  good  performers,  and  possible  ratings  creep  as 
threats  to  NSPS. 

"I  don't  know  how  I'll  fare  in  future  under  NSPS.  About  recruiting  in  NSPS,  NSPS  said  it  will  be  easier. 
My  colleagues  say  it's  harder.  When  people  see  the  range,  with  no  steps,  they  negotiate  for  high  end." 

"NSPS  will  be  hindered  in  the  future  by  Ops  Tempo  and  a  limited  size  workforce." 

"If  we’re  all  under  one  umbrella,  it  will  work  well. " 

Employees 

"I  am  optimistic  for  NSPS  here.  I  think  this  will  help.  We  are  not  there  yet,  but  I  think  we  will  be.  It  will 
help  to  address  poor  performers.  But  we  have  also  lost  some  high  performers.  We  also  have  problems 

getting  good  supervisors  with  the  5%." 

"The  payback  will  be  worth  it  if  NSPS  will  be  allowed  to  continue.  Some  form  of  NSPS  has  to  stay. 

Don't  give  us  the  GS  system  back." 

Commanders 

Outlook  for  NSPS  Acceptance  by  the  Workforce 

Some  employees  stated  that  NSPS  has  not  yet  realized  its  full  effectiveness,  but  that  additional 
guidance,  leadership,  and  improved  understanding  would  help.  A  number  of  employees 
predicted  that  time  will  smooth  the  way  for  acceptance  of  NSPS,  while  other  employees  noted 
that  NSPS'  success  depends  on  meeting  conditions,  such  as  one  personnel  system  for  all  DoD 
and  sufficient  training  and  understanding.  Other  employees  cited  competing  pay  systems,  ops 
tempo,  a  change  of  administration,  and  bargaining  unit  court  battles  as  challenges  to  the 
success  of  NSPS.  A  few  employees  argued  that  NSPS  would  not  change  anything.  Some 
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supervisors  purported  that  the  workforce  would  give  NSPS  a  chance  if  they  could  better 
understand  the  rationale  for  decisions  made.  Pay  pool  panel  members  were  generally 
supportive  of  NSPS,  but  some  were  unsure  of  its  effect  to  date.  Some  PRAs  believe  that 
workforce  trust  has  been  recaptured  following  NSPS  implementation.  Other  PRAs  saw  NSPS  as 
a  means  to  remain  focused  on  mission. 

"If  there's  commitment  from  SecDefon  down,  take  this  report ...  from  other  places,  that  captures  the 
frustrations  and  views  at  our  level,  and  doing  something  about  it,  making  it  right,  transparency  and 
pay  per  performance.  It's  going  through  growing  pains,  and  no  one  likes  change.  But  if  it  goes  the  way 

it  is  and  nothing  is  done  were  going  to  have  problems." 

Employee 

"I  think  people  want  this  to  work.  They're  not  ready  to  form  picket  lines.  They  want  to  understand  it.  If 
their  questions  are  (not)  constructive,  it's  just  that  everyone's  nervous  about  the  impact  of  decisions 

made  on  what  we  don't  understand." 

Supervisor 

Leadership,  Managing  Performance,  and  NSPS 

Some  employees  offered  that  leadership  effectiveness  and  not  NSPS  is  the  real  solution  to 
addressing  marginal  performers.  Some  pay  pool  panel  members  were  cautiously  optimistic  that 
NSPS  can  eliminate  low  performers  if  used  properly.  Commanders  and  PRAs  identified  the 
results  of  a  pay-for-performance  system  to  be  a  more-focused  workforce  and  a  mechanism  to 
refocus  marginal  performers. 

"I  think  NSPS  will  be  a  good  thing  in  the  end  because  eventually  when  people  start  utilizing  it  the  way 

we're  trained,  the  people  who  deserve  a  2  will  get  a  2." 

"NSPS  is  not  the  fix  for  poor  leadership.  Leaders  still  have  to  do  the  'dirty  work.'  Supervisors  are  still 

responsible  for  dealing  with  poor  performers." 

Employees 

"It's  very  difficult  to  say  that  you  really  performed  better  as  a  result  of  NSPS.  We  are  still  trying  to 

figure  it  out." 

"I'm  a  proponent  of  NSPS.  Done  correctly,  it  is  getting  us  the  right  behavior." 

Pay  Pool  Panel  Members 

Sustainment 

Two  areas  from  the  sustainment  perspective  includes  financial  and  evaluation  processes.  A 
number  of  employees  questioned  NSPS'  capability  for  long-term  financial  sustainment.  Senior 
leadership  cautions  against  ratings  inflation  overtime. 

"When  is  the  bank  going  to  get  broken?  It  seems  like  there  is  an  inordinate  amount  of  money  being 
paid  out.  It  can't  sustain  itself,  can  it?  People  come  to  expect  these  raises  going  forward,  and  I'm  not 

sure  we  can  afford  it." 

"I  believe  that  the  integrity  and  validity  of  the  current  5-point  rating  system  will  be  hard  to  sustain. 
There  will  be  a  natural  tendency  for  rating  creep  to  the  5  point  level  and  detract  from  the  pay  for 

performance  concept." 

PRAs 

Outlook  and  Predictions 

There  are  diverse  perspectives  on  the  outlook  for  NSPS.  A  lack  of  clarity  for  career  progression, 
uncertain  pay  raises,  the  option  to  move  to  non-NSPS  organizations,  supervisors'  perceived 
shortcomings,  and  a  more  competitive  workplace  climate  are  predicted  to  impact  negatively 
upon  retention  and  job  satisfaction.  Some  cite  competing  pay  systems,  ops  tempo,  and  a 
change  of  administration  as  challenges  to  the  success  of  NSPS.  Others  offered  that  additional 
guidance,  leadership,  and  training  would  help  NSPS  realize  its  full  effectiveness.  Senior  leaders 
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were  cautiously  optimistic  that  NSPS  can  eliminate  poor  performers  and  generally  wished  to 
pursue  NSPS  to  maturity,  but  also  indicated  that  it  is  too  early  to  evaluate  the  effects  of  NSPS 
on  individual  and  organizational  performance. 

The  concern  about  perceived  disincentives  for  progression,  such  as  the  five  percent 
reassignment  pay  increase  limit,  is  significant  for  both  employees  and  managers.  With  cautious 
optimism,  leaders  predicted  that  NSPS  will  cultivate  a  more  focused  workforce  and  will  help 
eliminate  marginal  performers. 

"It  might  take  time.  Right  now  it's  hurting  the  organization  because  of  the  morale.  Will  take  several 

years  and  good  leadership." 

"I  don't  think  NSPS  will  change  the  climate....  I  initially  thought  it  would  be  a  good  way  to  bring  in 
good  talent  like  the  private  sector  does,  but  it's  not  different  from  any  rating  system." 

"It's  too  early  to  tell  on  how  it  will  affect  me  personally.  Most  people  are  resistant  to  change.  There 

will  be  kinks.  I'm  going  to  wait  and  see.  The  concept  is  good.  [Nods] 

Employees 

"It  will  help  employees  prioritize.  We  have  a  shrinking  workforce  with  an  increased  workload,  so  this 

is  very  important." 

"I  believe  that  NSPS  will  make  the  organization  better  in  the  future.  We  got  over  the  lack  of  trust  with 

the  workforce.  I  hope  we  don't  go  back  to  the  old  system." 

PRAs 


3.6.  Site  Document  Analysis 

The  visited  sites  provided  organizational  NSPS-related  documents  to  the  visit  teams  prior  to  and 
during  the  visits.  The  documents  received  or  referenced  on  web  sites  are  not  necessarily  a 
comprehensive  indication  of  all  the  NSPS  documentation  these  organizations  prepared,  nor  do 
we  know  how  all  of  the  documentation  was  used.  Analysis  of  documentation  indicates  which 
topic  areas  were  most  frequently  available  for  information  and  use  by  management  and  the 
workforce. 

The  majority  of  documents  provided  were  pay  pool-related.  All  visited  organizations  had  pay¬ 
setting  policies  and  procedures  with  share  and  payout  distribution  rules.  Virtually  all  had  pay 
pool  panel  business  rules  that  also  applied,  where  relevant,  to  sub-pay  pool  panels.  Most  listed 
guidance  for  publication  of  the  organizational  rating  and  payout  results.  The  roles  of  the  PRA, 
pay  pool  manager,  and  pay  pool  members  were  the  next  most  available  documents,  followed 
closely  by  reconsideration  policy,  milestones  and  timelines,  the  identification  of  organizational 
pay  pools,  and  end-of-cycle  guidance.  The  least  frequently  observed  documents  were  the 
identity  of  pay  pool  panel  members  and  discretionary  pay  rules  (EPI  and  OAR).  Other 
documents  found  only  at  selected  sites  included  the  rating  and  payout  feedback  briefs  to  the 
workforce;  a  PAA  step-by-step  guide  for  preparing  the  performance  plan,  interim  review,  self- 
assessment  and  final  assessment;  and  a  guide  to  preparing  for  the  performance  management 
cycle. 

3.7.  Lessons  Learned;  Implementation  Teams  and  Pay  Pool  Panels 

Lessons  learned  were  provided  by  organization  implementation  teams  and  pay  pool  panels, 
both  consisting  primarily  of  senior  leadership.  Views  ranged  from  the  tactical  to  the  strategic. 
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3.7.1.  Implementation  Teams 

Timing 

Start  the  implementation  preparation  process  early  -  the  required  time  will  be  longer  than 
anticipated. 

Training 

Prepare  the  implementation  team,  human  resources  staff,  and  pay  pool  panel  members  with 
advance  training.  Provide  in-house  Personnel  Office  training  and  team  trainers  on  PAA,  NSPS, 
and  job  objective-writing  course  with  at  least  two  classes  per  week.  Ensure  active  attention  to 
filling  all  available  training  seats.  Tailor  a  standard  set  of  job  objectives  for  various  work 
fields/functional  areas  using  SME  teams. 

"The  training  was  useful  in  that  it  trained  us  in  the  things  we  were  going  to  be  doing ...  performance 
management,  pay  pool....  The  problem  was  the  gaps;  we  needed  separate  training  on  setting  up  the 

implementation  team." 

Implementation  Team  Member 


Structure.  Staffing,  and  Processes 

Implementation  team  composition  should  include  human  resources,  representatives  from  the 
Office  of  the  Comptroller  and  of  the  General  Counsel,  a  strategic  planner,  and  a  statistician. 
Ensure  that  one  person  is  dedicated  to  system  testing  and  the  conversion  process.  Ensure  a 
dedicated,  full-time  Transition  Manager  with  decision-making  authority  to  head 
implementation  efforts.  Consider  establishing  an  Executive  Steering  Committee  to  make 
implementation  decisions.  During  conversion  ensure  1)  better  definition  in  the  classification 
process,  2)  standardized  position  descriptions,  and  3)  clear  guidance  on  bringing  employees 
into  NSPS  from  other  systems.  Assemble  ad  hoc  specialty  teams  as  needed.  Have 
implementation  team  liaisons  at  the  unit  level. 

"I  would  recommend  that  the  transition  manager  is  a  decision  maker,  not  just  an  HR  person. 

You  need  senior  leadership  to  push  this  forward." 

Implementation  Team  Member 


Support 

Enlist  senior  leadership's  continuous  and  visible  support  and  include  a  focus  on  military 
supervisors  to  support  NSPS.  Learn  from  earlier  spirals  or  demonstration  projects  how  to  best 
transition.  Establish  a  full-time  PAA  help  desk  or  PAA  technical  team  to  assist  users.  Budget 
sufficiently  for  training  and  materials.  Do  not  do  a  reorganization  during  the  transition  to  NSPS. 

Policy  Development 

Publish  a  compensation  philosophy  early;  start  conservatively  with  pay  setting  and  pay  pool 
policies.  Consider  installation-wide  focus  groups  to  canvass  concerns  and  questions. 

"The  Technical  Director  wanted  to  make  sure  that  the  well-defined  organizational  strategy  was  not 
lost  in  the  transition;  that's  why  we  got  so  much  support.  In  our  sister  organization  it  was  run  much 
more  from  the  HR  side....  The  problem  with  that  is  that  they  lost  opportunities  to  use  the  change  into 
NSPS  as  an  opportunity  to  exercise  strategic  intent,  like  rewarding  supervisors ...  control  points  and  if 

you  complete  this  training  you'll  get  more  money." 

Implementation  Team  Member 

Communication 

Communicate  to  employees  and  supervisors  often,  with  consistent  and  pertinent  information, 
to  ensure  effective  understanding  and  buy-in.  Publish  all  information  -  especially 
Implementing  Issuances,  implementation  regulations,  PAA  links,  and  milestone  plans.  Use  a 
variety  of  methods  and  media:  town  halls;  one-on-one  sessions;  Commander's  Calls; 
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information  dissemination  through  the  steering  committee;  intranet  or  Share  Point  web  site 
with  local  command,  higher  command,  and  DoD  documents,  links,  and  video;  weekly 
newsletters;  published  leadership  update  notes;  emails;  and  an  NSPS  library.  Scan  and  send 
anything  not  received  in  soft  copy.  Ensure  supervisors  are  aware  of  NSPS's  advantages  of 
advancement  potential  and  flexibilities. 

"Communication  to  set  expectations  is  key  -  especially  as  we  get 
new  employees  and  people  and  different  roles." 

Pay  Pool  Panel  Member 

"Communicate  to  the  employees.  While  I  had  some  town  halls  and  sent  out  some  emails,  a  lot  of  that 
(communication)  was  done  at  the  management  level.  We  need  to  communicate  more." 

Implementation  Team  Member 


Conversion  Issues 

Difficulties  occurring  during  the  conversion  process  included:  1)  pay  anomalies,  especially  with 
Pay  Band  YA2,  and  pay  compression;  2)  expectations  of  GS  employees  related  to  pay  and  step 
increases;  3)  defining  who  is  a  supervisor  after  eliminating  "leads"  upon  conversion  to  NSPS;  4) 
transitioning  the  financial  systems  for  working  capital  fund  organizations;  and  5)  converting 
employees  into  the  new  system. 

"We  needed  more  money  to  cover  training,  create  materials,  etc.  [Much  agreement], 

and  money  is  scarce." 

Implementation  Team  Member 


3.7.2.  Pay  Pool  Panels 

Training 

Hold  a  mock  pay  pool  panel  during  implementation  and  for  sustainment  training  of  new  pay 
pool  panel  members.  Make  the  pay  pool  panel  manager  a  senior  civilian.  Roles  important  to  the 
success  of  the  pay  pool  panel  process  include  1)  pay  pool  administrator,  2)  advisors,  3) 
executive  leadership,  and  4)  implementation  team. 

"I  was  not  satisfied  not  having  a  civilian  as  a  pay  pool  manager.  I  would  change  that.  It  worked  at 

that  time,  but  now  I  would  consider  that.  I  think  that's  the  right  answer." 

PRA 


Structure,  Staffing,  and  Processes 

Factors  to  consider  in  composing  pay  pools  and  sub-pay  pools  include  1)  size  of  the 
organization,  2)  organizational  structure,  3)  budget,  4)  functional  roles/occupational  structure, 
5)  mixture  of  civilian  and  military  pay  pool  panel  members. 

"Headquarters  is  going  to  publish  broad  guidance  on  establishing  a  compensation  philosophy.  We 
went  headlong  into  this  thing  without  any  of  that.  This  is  a  lesson  learned.  We  now  have  visibility  on 

salary  growth.  This  wasn't  clearly  defined  at  first." 

Commander 


Support 

Involve  senior  leadership  early  on  in  providing  guidance  to  the  pay  pool  panel. 

"You  need  to  have  someone  at  a  high  level  in  your  organization  take  on  the  lead  role.  When  the  policy 
comes  from  them,  the  guidance  is  not  questioned.  It  eliminates  the  second  guessing." 

Commander 
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3. 7.3.  Summary  of  Lessons  Learned 

There  are  three  essential  areas  that  require  attention  in  preparing  for  NSPS  implementation: 
training;  establishing  the  appropriate  structure,  staffing,  and  processes  for  implementation 
teams  and  pay  pool  panels;  and  establishing  internal  support  through  senior  leadership 
endorsement,  as  well  as  technical  support  for  NSPS  automated  processes.  Additional  key 
processes  included  effective  communication  up  and  down  the  chain  of  command;  defining 
team  processes;  initiating  required  actions  in  a  timely  manner;  planning  for  adequate  budget; 
and  avoiding  other  large-scale  organizational  activities,  such  as  reorganizations,  during  NSPS 
implementation. 
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4.  Analysis  of  SOFS-C  Data 


4.1.  Introduction  and  Method 

4.1.1.  Purpose 

This  chapter  summarizes  the  results  of  the  February  2008  Status  of  Forces  Survey-Civilian 
(SOFS-C).  The  purpose  of  SOFS-C  is  to  assess  the  attitudinal  posture  of  the  DoD  appropriated 
fund  civilian  workforce.  It  considers  DoD  human  capital  matters,  NSPS,  and  other  topics  of 
special  interest.  The  Defense  Manpower  Data  Center  (DMDC)  administers  it  yearly. 

4.1.2.  Rationale  for  Use  in  This  NSPS  Report 

The  NSPS  Evaluation  Plan  identifies  the  SOFS-C  as  one  of  the  primary  sources  of  attitudinal  data 
for  use  in  evaluating  NSPS 

■  Most  KPPs  link  to  several  evaluation  metrics 

■  Data  sources  for  many  of  the  evaluation  metrics  are  survey  responses 

■  Analyzing  these  responses  over  time  helps  track  workforce  attitudes  toward  NSPS 
implementation 

4.1.3.  SOFS-C  Section  and  Methodology 

This  chapter  presents  SOFS-C  results  broken  into  five  sections  that  are  applicable  to  NSPS 
evaluation.  They  are  Satisfaction  with  Job/Workplace/Coworkers;  Satisfaction  with 
Leadership/Supervisors/Policies;  Satisfaction  with  Performance  Management;  Satisfaction  with 
Career  Progression;  and  Attitudes  toward  NSPS. 

The  SOFS-C  uses  stratified  random  sampling  across  the  DoD  appropriated  fund  civilian 
workforce.  It  presents  demographic  reporting,  including  total  workforce  with  component 
breakout,  total  workforce  with  NSPS  spiral/non-spiral  breakout,  and  by  general  type  of  job.  It 
also  presents  consolidated  reporting  for  participants  in  NSPS  pay  bands,  as  well  as  by  spiral. 

4.1.4.  SOFS-C  Data  Reviewed  in  This  Report 

Data  from  the  February  2008  survey  are  used  wherever  possible.  As  of  the  publication  of  this 
report  February  2008  data  for  some  questions  in  the  "Attitudes  toward  NSPS"  section  were  not 
available.  Where  noted.  May  2007  survey  data  were  used  in  lieu  of  February  2008  data. 
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4.1.5.  Graphical  Representation  of  Results  and  Analysis  Methodology 

The  analysis  examines  the  frequencies  of  "favorable  response  rates";  specifically,  the  analysis 
seeks  to  determine  whether  favorable  response  rates  for  certain  subgroups  of  the  population 
are  different  from  the  overall  favorable  response  rate,  and  how  overall  and  subgroup  responses 
change  overtime. 

"Favorable  response  rate"  is  defined  as  follows:  For  each  survey  item,  as  appropriate,  the 
analysis  collapsed  the  "positive"  and  "strongly  positive"  responses  to  a  given  survey  item  into 
one  "favorable"  category.  The  "negative"  and  "strongly  negative"  responses  were  collapsed 
into  a  single  "unfavorable"  category,  leaving  a  "neutral"  category  in  between. 

Most  of  the  following  pages  show  one  or  more  sets  of  two  graphs  each.  The  graphs  on  the  left 
show  the  "favorable"  responses  to  a  given  survey  item,  both  total  DoD  (NSPS  and  non-NSPS) 
and  by  NSPS  spiral.  The  graphs  on  the  right  show  the  negative  responses  to  that  same  item.  In 
most  cases,  the  scales  are  the  same  within  each  set  of  graphs  to  allow  easy  comparison  of 
favorable  and  unfavorable  results.  However,  to  improve  readability,  some  pairs  of  figures  are 
shown  with  different  scales  for  the  responses.  These  are  noted  in  each  case. 

Summary  comments  are  located  at  the  beginning  of  each  section.  In  addition,  some  comments 
appear  under  a  set  of  graphs  to  point  out  themes  that  emerge  from  the  analysis. 

4.2.  Satisfaction  with  Job.  Work,  the  Workplace,  and  Coworkers 

Figures  4.1  and  4.2:  Considering  everything,  how  satisfied  are  you  with  your  job? 

Figures  4.3  and  4.4:  Agree/Disagree:  I  like  the  kind  of  work  I  do. 

Figures  4.5  and  4.6:  Agree/Disagree:  My  work  gives  me  a  feeling  of  personal  accomplishment. 
Figures  4.7  and  4.8:  Agree/Disagree:  The  work  I  do  is  important. 

Figures  4.9  and  4.10:  Agree/Disagree:  I  know  how  my  work  relates  to  the  Agency's  goals  and 
priorities. 

Figures  4.11  and  4.12:  Agree/Disagree:  The  people  I  work  with  cooperate  to  get  the  job  done. 

Figures  4.13  and  4.14:  Agree/Disagree:  My  work  unit  is  able  to  recruit  people  with  the  right 
skills. 

Figures  4.15  and  4.16:  Agree/Disagree:  Employees  have  a  feeling  of  personal  empowerment 
with  respect  to  work  processes. 

Summary: 

Spirals  1.1  and  1.2  track  very  closely  with  the  overall  DoD  population  on  most  of  the  job  and 
work  satisfaction  measures  (Figures  4.1  through  4.10).  While  Spiral  1.1  frequently  debuted  at  a 
higher  level  of  "favorable  response"  than  the  overall  DoD  population  and  then  dropped  to  a 
level  slightly  below  that  of  the  overall  DoD  population,  favorable  response  rates  for  Spiral  1.1  in 
these  areas  remain  high,  and  close  to  that  of  the  overall  DoD  population.  Overall,  there  has 
been  very  little  change  in  favorable  or  unfavorable  responses  to  these  job  and  work  satisfaction 
items  for  both  the  DoD  population  and  the  NSPS  spirals. 

All  NSPS  spirals  track  closely  with  the  overall  DoD  population  on  measures  concerning 
coworkers  and  workplace  environment  as  well.  All  NSPS  spirals  have  remained  at  a  higher  level 
of  favorable  response  than  the  overall  DoD  population  on  "the  people  I  work  with  cooperate  to 
get  the  job  done"  (Figure  4.11).  While  all  NSPS  spirals  remain  at  a  higher  level  of  favorable 
response  than  the  overall  DoD  population  on  "my  work  unit  is  able  to  recruit  people  with  the 
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right  skills"  (Figure  4.13),  there  is  also  a  somewhat  high  level  of  unfavorable  response  among  all 
groups  to  this  item,  at  around  25%.  There  is  also  a  somewhat  high  level  of  unfavorable 
responses  to  "employees  have  a  feeling  of  personal  empowerment"  (Figure  4.16)  at  around 
25%  for  all  groups,  but  again,  there  has  been  little  volatility  over  time  with  either  the  favorable 
or  the  unfavorable  responses  to  these  items. 


Overall,  how  satisfied  are  you  with  your  job?  - 
Satisfied 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  DoD  ♦  Spiral  1.1 

—♦—Spiral  1.2  — ■ —  Spiral  1.3 


Overall,  how  satisfied  are  you  with  your  job?  - 
Dissatisfied 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  DoD  ♦  Spiral  1.1 

—♦—Spiral  1.2  — ■ —  Spiral  1.3 


Figures  4.1  and  4.2  -  Overall  Job  Satisfaction 

Overall  job  satisfaction  is  high  among  all  groups.  While  Spiral  1.1  dropped  below  the  overall  DoD 
population  in  favorable  response  since  May  2006,  it  rose  slightly  in  February  2008. 


I  like  the  kind  of  work  I  do  -  Agree 


I  like  the  kind  of  work  I  do  -  Disagree 
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Figures  4.3  and  4.4  -  Work  Satisfaction 


NSPS  spirals  track  closely  with  the  overall  DoD  population. 
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My  work  gives  me  a  feeling  of  personal 
accomplishment  -  Agree 
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My  work  gives  me  a  feeling  of  personal 
accomplishment  -  Disagree 
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Figures  4.5  and  4.6  -  Feeling  of  Accomplishment 
NSPS  spirals  track  closely  with  the  overall  DoD  population. 
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The  work  I  do  is  important  -  Agree 
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Figures  4.7  and  4.8-  Work  Importance 
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NSPS  spirals  track  closely  with  the  overall  DoD  population. 
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I  know  how  my  work  relates  to  my  organization's 
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goals  and  priorities  -  Agree 
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Figures  4.9  and  4.19  -  Relation  to  Agency's  Goals 

Spiral  1.2  remains  at  a  higher  level  of  favorable  response  compared  to  the  overall  DoD  population. 


The  people  I  work  with  cooperate  to  get  the  job 
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Figures  4.11  and  4.12  -  People  Cooperate 
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NSPS  spirals  track  more  favorably  than  the  overall  DoD  population  on  cooperation  to  get  the  job  done. 
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My  work  unit  is  able  to  recruit  people  with  the  right 


My  work  unit  is  able  to  recruit  people  with  the  right 
skills  -  Disagree 
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Figures  4.13  and  4.14  -  Recruiting  People 


■Spiral  1.1 
■Spiral  1.3 


While  all  NSPS  spirals  remain  at  a  higher  level  of  favorable  response  than  the  overall  DoD  population, 
there  is  also  a  somewhat  high  level  of  unfavorable  response. 


Employees  have  a  feeling  of  personal 
empowerment  with  regard  to  work  processes  - 
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Employees  have  a  feeling  of  personal 
empowerment  with  regard  to  work  processes 
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Figures  4.15  and  4.16-  Personal  Empowerment 


Spiral  1.2  has  consistently  tracked  more  favorably  on  this  item  than  the  overall  DoD  population. 
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4.3.  Satisfaction  with  Leadership.  Supervisors,  and  Policies 

Figures  4.17  and  4.18:  How  satisfied  are  you  with  the  policies  and  practices  of  your  senior 
leaders? 

Figures  4.19  and  4.20:  How  satisfied  are  you  with  the  information  you  receive  from 
management  on  what's  going  on  in  your  organization? 

Figures  4.21  and  4.22  Agree/Disagree:  Managers  communicate  the  goals  and  priorities  of  the 
organization. 

Figures  4.23  and  4.24:  Agree/Disagree:  Managers  review  and  evaluate  the  organization's 
progress  toward  meeting  its  goals  and  objectives. 

Figures  4.25  and  4.26:  Agree/Disagree:  In  my  organization,  leaders  generate  high  levels  of 
motivation  and  commitment  in  the  workforce. 

Figures  4.27  and  4.28:  Overall,  how  good  a  job  do  you  feel  is  being  done  by  your  immediate 
supervisor/team  leader? 

Figures  4.29  and  4.30:  Agree/Disagree:  I  have  a  high  level  of  respect  for  my  organization's 
senior  leaders. 

Figures  4.31  and  4.32:  Agree/Disagree:  I  have  trust  and  confidence  in  my  supervisor. 

Figures  4.33  and  4.34:  Agree/Disagree:  Discussions  with  my  supervisor/team  leader  about  my 
performance  are  worthwhile. 

Summary: 

All  NSPS  groups  track  closely  (within  a  few  percentage  points)  to  the  overall  DoD  population  on 
satisfaction  with  leadership,  supervisors,  and  policies.  All  NSPS  groups  have  more  favorable 
responses  to  "managers  communicate  organizational  goals"  and  "managers  review 
organizational  progress"  than  the  overall  DoD  population,  although  there  has  been  little 
movement  in  attitudes  over  time  (Figures  4.21  through  4.24).  Most  NSPS  groups  have  more 
favorable  and  less  unfavorable  responses  to  "leaders  generate  high  levels  of  motivation" 
(Figures  4.25  and  4.26).  However,  all  groups  had  somewhat  high  levels  of  unfavorable 
responses  to  this  item,  at  around  30%. 

Most  NSPS  groups  track  closely  to  the  overall  DoD  population  on  "having  a  high  level  of  respect 
for  my  organization's  senior  leaders"  (Figures  4.29  and  4.30),  although  Spiral  1.2  tracks  slightly 
above  the  overall  DoD  population  in  favorable  responses  for  this  item.  Figures  4.31  through 
4.34  show  that  all  NSPS  groups  track  very  closely  with,  if  slightly  more  favorably  than,  the 
overall  DoD  population  on  having  "trust  and  confidence  in  my  supervisor"  and  believing  that 
"discussions  with  my  supervisor  about  my  performance  are  worthwhile." 
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4.  Analysis  of  SO FS-C  Data 


NS  PS  Spiral  1  Evaluation  Report 


How  satisfied  are  you  with  the  policies  and  practices 
of  your  senior  leaders?  -  Satisfied 


Total  DoD 


•Spiral  1.1 


How  satisfied  are  you  with  the  policies  and  practices 
of  your  senior  leaders?  -  Dissatisfied 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 


Total  DoD 
Spiral  1.2 


-♦-Spiral  1.2  — ■ — Spiral  1.3 

Figures  4.17  and  4.18  -  Leaders'  Policies  and  Practices 

NSPS  spirals  track  closely  with  the  overall  DoD  population. 


■Spiral  1.1 
•Spiral  1.3 


How  satisfied  are  you  with  the  information  you 
receive  from  management  on  what  is  going  on  in 


How  satisfied  are  you  with  the  information  you 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  DoD  —♦—Spiral  1.1 

-♦-Spiral  1.2  — ■ —  Spiral  1.3 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

-A—  Total  DoD  —♦—Spiral  1.1 

—♦—Spiral  1.2  Spiral  1.3 

Figures  4.19  and  4.20  -  Management  Information 


Spiral  1.2  has  consistently  tracked  more  favorably  on  this  item  than  the  overall  DoD  population. 
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Managers  communicate  the  goals  and  priorities  of 
the  organization  -  Agree 


Total  DoD 
-Spiral  1.2 


■Spiral  1.1 
■Spiral  1.3 


Managers  communicate  the  goals  and  priorities  of 
the  organization  -  Disagree 


70% 

60% 

50% 

40% 

30% 

20% 

10% 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 


Total  DoD 
-Spiral  1.2 


■Spiral  1.1 
•Spiral  1.3 


Figures  4.21  and  4.22  -  Goals/Priorities  Communicated 
All  NSPS  spirals  are  slightly  more  favorable  than  the  overall  DoD  population. 


Managers  review  and  evaluate  the  organization's 
progress  towards  meetings  its  goals  and  obligations  - 


Managers  review  and  evaluate  the  organization's 
progress  towards  meetings  its  goals  and  obligations  - 

Disagree 

70% 

60% 

50% 

40% 

30% 

20% 


10% 

Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 


Total  DoD  —♦—Spiral  1.1  -A— Total  DoD 

-Spiral  1.2  Spiral  1.3  —♦—Spiral  1.2 

Figures  4.23  and  4.24  -  Organization's  Progress 


-  Spiral  1.1 
Spiral  1.3 


NSPS  groups  have  had  slightly  more  favorable  responses  to  this  item  than  the  overall  DoD  population 
over  time. 
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NS  PS  Spiral  1  Evaluation  Report 


Leaders  generate  high  levels  of  motivation  and 
commitment  in  the  workforce  -  Agree 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  DoD  —♦—Spiral  1.1 

-♦-Spiral  1.2  — ■ — Spiral  1.3 


Leaders  generate  high  levels  of  motivation  and 
commitment  in  the  workforce  -  Disagree 


Total  DoD 
-Spiral  1.2 


-Spiral  1.1 
-Spiral  1.3 


Figures  4.25  and  4.26  -  Leaders  Motivate 


Spiral  1.2  has  had  significantly  higher  favorable  responses  and  lower  unfavorable  responses  to  this  item 
than  the  overall  DoD  population. 


Overall,  how  good  a  job  is  being  done  by  your 
immediate  supervisor/team  leader?  -  Good  Job 


Overall,  how  good  a  job  is  being  done  by  your 
immediate  supervisor/team  leader?  -  Poor  Job 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 


Total  DoD 
-Spiral  1.2 


-Spiral  1.1 
-Spiral  1.3 


Total  DoD 
-Spiral  1.2 


1.1 

1.3 


Figures  4.27  and  4.28  -  Supervisor's  Effectiveness 


NSPS  spirals  track  closely  with  the  overall  DoD  population. 
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I  have  a  high  level  of  respect  for  my  Organization's 
senior  leaders- Agree 


Total  DoD 
-Spiral  1.2 


-Spiral  1.1 
Spiral  1.3 


I  have  a  high  level  of  respect  for  my  Organization's 
senior  leaders  -  Disagree 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  —♦—Spiral  1.1 


-Spiral  1.2 


-Spiral  1.3 


Figures  4.29  and  4.30  -  Respect  for  Senior  Leaders 

Spiral  1.2  has  had  significantly  higher  favorable  responses  to  this  item  than  the  overall  DoD  population. 


I  have  trust  and  confidence  in  my  supervisor  - 
Agree 


I  have  trust  and  confidence  in  my  supervisor  - 
Disagree 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

-A— Total  DoD  —♦—Spiral  1.1 

—♦—Spiral  1.2  Spiral  1.3  —♦—Spiral  1.2 

Figures  4.31  and  4.32  -  Trust  in  Supervisor 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  —♦—Spiral  1.1 


•Spiral  1.3 


NSPS  groups  track  slightly  higher  than  the  overall  DoD  population  on  this  item. 
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NS  PS  Spiral  1  Evaluation  Report 


Discussions  with  my  supervisor  about  my 


Discussions  with  my  supervisor  about  my 

65% 

60% 

55% 

50% 

45% 

40% 

35% 

30% 

25% 

20% 

15% 

10% 

Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 


Total  DoD 
-Spiral  1.2 


-Spiral  1.1 
-Spiral  1.3 


Total  DoD 
-Spiral  1.2 


-Spiral  1.1 
-Spiral  1.3 


Figures  4.33  and  4.34  -  Performance  Discussions 


Spiral  1.1's  favorable  response  rate  fell  from  May  2006  to  May  2007,  but  rose  slightly  in  February  2008. 


4.4.  Satisfaction  with  Performance  Management 

Figures  4.35  and  4.36:  How  satisfied  are  you  with  the  recognition  you  receive  for  doing  a  good 
job? 

Figures  4.37  and  4.38:  Agree/Disagree:  Creativity  and  innovation  are  rewarded. 

Figures  4.39  and  4.40  Agree/Disagree:  Differences  in  performance  are  recognized  in  a 
meaningful  way. 

Figures  4.41  and  4.42:  Agree/Disagree:  In  my  work  unit,  steps  are  taken  to  deal  with  a  poor 
performer  who  cannot  or  will  not  improve. 

Figures  4.43  and  4.44:  Agree/Disagree:  My  performance  appraisal  is  a  fair  reflection  of  my 
performance. 

Figures  4.45  and  4.46:  Agree/Disagree:  In  my  most  recent  performance  appraisal,  I  understood 
what  I  had  to  do  to  be  rated  at  different  performance  levels. 

Figures  4.47  and  4.48:  Agree/Disagree:  Pay  raises  depend  on  how  well  employees  perform  their 
jobs. 

Figures  4.49  and  4.50:  Considering  everything,  how  satisfied  are  you  with  your  pay? 

Summary: 

As  of  February  2008,  all  NSPS  groups  were  slightly  more  satisfied  with  the  recognition  they 
receive  for  doing  a  good  job  than  the  overall  DoD  population  (Figures  4.35  and  4.36).  All  NSPS 
groups  also  had  slightly  more  favorable  responses  to  "Creativity  and  innovation  are  rewarded" 
(Figures  4.37  and  4.38). 

Spiral  1.1  showed  an  uptick  in  February  2008  in  its  favorable  responses  to  "differences  in 
performance  are  recognized  in  a  meaningful  way"  (Figures  4.39  and4.40),  and  all  other  NSPS 
groups  were  more  favorable  than  the  overall  DoD  population.  However,  all  groups  had  almost 
as  many  unfavorable  responses  as  they  did  favorable  responses  to  this  item.  Similarly,  all  NSPS 
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NSPS  Spiral  1  Evaluation  Report 


4.  Analysis  ofSOFS-C  Data 


groups  were  less  unfavorable  than  the  overall  DoD  population  on  "dealing  with  poor 
performers"  (Figures  4.41  and  4.42)  but  all  groups  had  more  unfavorable  than  favorable 
responses  to  this  item. 

Spirals  1.1,  1.2,  and  1.3  have  shown  a  decline  over  time  compared  to  the  overall  DoD 
population  in  favorable  responses  regarding  performance  appraisals  (Figures  4.43  and  4.45), 
although  Spiral  1.1  has  shown  a  slight  uptick  in  favorable  responses  to  these  items  after  the 
close  of  its  second  cycle  in  NSPS. 

All  NSPS  spirals  agree  much  more  strongly  than  the  rest  of  the  DoD  population  that  their  pay 
raises  depend  on  job  performance  (Figure  4.47),  while  levels  of  satisfaction  and  dissatisfaction 
with  pay  are  similar  for  those  within  and  outside  of  NSPS  (Figures  4.49  and  4.50). 


How  satisfied  are  you  with  the  recognition  you 
receive  for  doing  a  good  job? 

Satisfied 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  —♦—Spiral  1.1 


-Spiral  1.2 


Spiral  1.3 


How  satisfied  are  you  with  the  recognition  you 
receive  for  doing  a  good  job? 
Dissatisfied 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  —♦—Spiral  1.1 


-Spiral  1.2 


-Spiral  1.3 


Figures  4.35  and  4.36  -  Recognition  for  Performance 


Spiral  1.1's  satisfaction  rose  in  February  2008  after  their  second  payout,  after  dropping  in  May  2007. 
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NS  PS  Spiral  1  Evaluation  Report 


Creativity  and  innovation  are  rewarded  -  Agree 


Creativity  and  innovation  are  rewarded  -  Disagree 


Total  DoD 
■Spiral  1.2 


■Spiral  1.1 
■Spiral  1.3 


Total  DoD 
-Spiral  1.2 


■Spiral  1.1 
■Spiral  1.3 


Figures  4.37  and  4.38  -  Creativity  and  Innovation  are  Rewarded 


All  NSPS  groups  have  higher  favorable  response  rates  and  lower  unfavorable  response  rates  than  the 
overall  DoD  population. 


Differences  in  performance  are  recognized  in  a 
meaningful  way  -  Agree 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  Spiral  1.1 


■Spiral  1.2 


■Spiral  1.3 


Differences  in  performance  are  recognized  in  a 
meaningful  way  -  Disagree 


Total  DoD 
-Spiral  1.2 


■Spiral  1.1 
■Spiral  1.3 


Figures  4.39  and  4.40  -  Performance  Differences 


Spiral  1.1's  responses  to  this  item  improved  in  February  2008,  after  their  second  rating  cycle  and 
performance  payout. 
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In  my  work  unit,  steps  are  taken  to  deal  with  a  poor 
performer  who  cannot  or  will  not  improve  - 


In  my  work  unit,  steps  are  taken  to  deal  with  a  poor 
performer  who  cannot  or  will  not  improve  - 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
-±—  Total  DoD  —♦—Spiral  1.1 

—♦—Spiral  1.2  — ■ —  Spiral  1.3  “♦" 

Figures  4.41  and  4.42  -  Poor  Performers 


Total  DoD 
-Spiral  1.2 


-Spiral  1.1 
-Spiral  1.3 


NSPS  groups  have  higher  favorable  response  rates  than  unfavorable  response  rates  to  this  item,  while 
the  overall  DoD  population  does  not. 


My  performance  appraisal  is  a  fair  reflection  of  my 
performance  -  Agree 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 

Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 


My  performance  appraisal  is  a  fair  reflection  of  my 
performance  -  Disagree 


Oct- 04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 


Total  DoD 
-Spiral  1.2 


-Spiral  1.1 
-Spiral  1.3 


Figures  4.43  and  4.44  -  Fair  Appraisal 


Total  DoD 
Spiral  1.2 


-Spiral  1.1 
-Spiral  1.3 


All  NSPS  spirals  declined  on  this  measure  through  their  first  payouts  under  NSPS.  However,  Spiral  1.1 
became  slightly  more  favorable  in  February  2008  after  its  second  payout. 
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NS  PS  Spiral  1  Evaluation  Report 


In  my  most  recent  performance  appraisal,  I 
understood  what  I  had  to  do  to  be  rated  at 


In  my  most  recent  performance  appraisal,  I 
understood  what  I  had  to  do  to  be  rated  at 
different  levels  -  Disagree 


Total  DoD  —♦—Spiral  1.1  -A—  Total  Do D 

-Spiral  1.2  — ■ —  Spiral  1.3  —♦—Spiral  1.2 

Figures  4.45  and  4.46  -  Understood  Different  Levels 


-Spiral  1.1 
-Spiral  1.3 


All  NSPS  spirals  declined  on  this  measure  through  their  first  payouts  under  NSPS.  However,  Spiral  1.1 
became  slightly  more  favorable  in  February  2008  after  its  second  payout. 


Pay  raises  depend  on  how  well  employees  perform 
their  jobs-  Agree 


Pay  raises  depend  on  how  well  employees  perform 
their  jobs  -  Disagree 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

-A— Total  DoD  —♦—Spiral  1.1 

—♦—Spiral  1.2  Spiral  1.3  —♦—Spiral  1.2 

Figures  4.47  and  4.48  -  Pay  Raise  and  Performance 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  —♦—Spiral  1.1 


-Spiral  1.3 


NSPS  Spirals  1.1,  1.2,  and  1.3  are  substantially  more  favorable  on  this  item  than  the  overall  DoD 
population.  All  groups  have  increased  in  favorable  responses  since  November  2006. 
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Considering  everything,  how  satisfied  are  you  with 
your  pay? -Satisfied 


Considering  everything,  how  satisfied  are  you  with 
your  pay?  -  Dissatisfied 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  —♦—Spiral  1.1 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  —♦—Spiral  1.1 


■Spiral  1.2  -■—Spiral  1.3  —♦—Spiral  1.2 

Figures  4.49  and  4.50  -  Pay  Satisfaction 


Spiral  1.3 


Despite  the  sharp  initial  decline  for  Spiral  1.1,  all  NSPS  groups  track  consistently  with  the  overall  DoD 
population  on  pay  satisfaction. 


4.5.  Satisfaction  with  Career  Progression  Opportunities 

Figures  4.51  and  4.52:  Agree/Disagree:  I  am  given  a  real  opportunity  to  improve  my  skills  in  my 
organization. 

Figures  4.53  and  4.54:  How  satisfied  are  you  with  your  opportunities  to  get  a  better  job  in  your 
organization? 

Figures  4.55  and  4.56  Agree/Disagree:  Promotions  in  my  work  unit  are  based  on  merit. 

Summary: 

There  is  very  little  difference  between  NSPS  spirals  and  the  overall  DoD  population  on  their 
responses  to  "being  given  a  real  opportunity  to  improve  my  skills"  and  "satisfaction  with 
opportunities  to  get  a  better  job  in  my  organization"  (Figures  4.51  through  4.54).  All  groups 
have  approximately  equal  favorable  and  unfavorable  responses  to  "satisfaction  to  get  a  better 
job  in  my  organization." 

Figures  4.55  and  4.56  show  all  NSPS  groups  with  more  favorable  and  less  unfavorable  responses 
to  "promotions  in  my  work  unit  are  based  on  merit."  Spiral  1.2's  favorable  response  in 
February  2008  was  approximately  44%  compared  to  the  overall  DoD  favorable  response  of 
approximately  34%. 
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NS  PS  Spiral  1  Evaluation  Report 


I  am  given  a  real  opportunity  to  improve  my  skills  - 
Agree 


I  am  given  a  real  opportunity  to  improve  my  skills  - 
Disagree 


Oct- 04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08  Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

-±—  Total  DoD  —♦—Spiral  1.1  -A—  Total  DoD  —♦—Spiral  1.1 

—♦—Spiral  1.2  Spiral  1.3  —♦—Spiral  1.2  Spiral  1.3 

Figures  4.51  and  4.52  -  Skills  Improvement 

Spiral  1.1's  favorable  response  rate  increased  slightly  in  February  2008  after  falling  between  May  2006 
and  May  2007. 


How  satisfied  are  you  with  your  opportunity  to  get  a 
better  job  in  your  organization?  -  Satisfied 


50% 

45% 

40% 

35% 

30% 

25% 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
Total  DoD  —♦—Spiral  1.1 


■Spiral  1.2 


How  satisfied  are  you  with  your  opportunity  to  get  a 
better  job  in  your  organization?  -  Dissatisfied 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

-A— Total  DoD  —♦—Spiral  1.1 

-■—Spiral  1.3  -♦—Spiral  1.2  Spiral  1.3 

Figures  4.53  and  4.54  -  Opportunity  for  Better  Job 


All  groups  have  approximately  equal  favorable  and  unfavorable  response  rates  to  this  item. 
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Promotions  in  my  work  unit  are  based  on  merit  - 


Promotions  in  my  work  unit  are  based  on  merit  - 
Disagree 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Spiral  1.1 
Spiral  1.3 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  DoD  —♦—Spiral  1.1 

—♦—Spiral  1.2  Spiral  1.3 


Total  DoD 
-Spiral  1.2 


Figures  4.55  and  4.56  -  Merit-Based  Promotions 

NSPS  groups  are  significantly  more  favorable  on  this  measure  than  the  overall  DoD  population. 


4.6.  Attitudes  about  NSPS 

Figures  4.57  through  4.59:  Agree/Disagree:  NSPS  will  improve/has  improved  hiring  new 
employees. 

Figures  4.60  through  4.62:  Agree/Disagree:  NSPS  will  improve/has  improved 
disciplining/correcting  poor  work  performance. 

Figures  4.63  through  4.65:  Agree/Disagree:  NSPS  will  improve/has  improved  rewarding  good 
work  performance. 

Figures  4.66  through  4.68:  Agree/Disagree:  NSPS  will  improve/has  improved  linking  pay  to 
performance. 

Figures  4.69  through  4.71:  Agree/Disagree:  NSPS  will  improve/has  improved  classification  of 
jobs  by  series  and  pay  grade/pay  band. 

Figures  4.72  through  4.74:  Agree/Disagree:  NSPS  will  improve/has  improved  communication 
between  supervisors  and  employees. 

Figures  4.75  through  4.77:  Agree/Disagree:  NSPS  will  improve/has  improved  ensuring  individual 
performance  supports  organizational  mission  effectiveness. 

Figure  4.78:  For  employees  in  your  organization,  how  does  NSPS  compare  to  your  previous 
personnel  system  for...  [includes  only  "better/much  better"  responses]. 

Figures  4.79  through  4.80:  NSPS  is  better/worse  than  the  previous  system  for  Hiring, 
Placement,  and  Promotion. 

Figures  4.81  and  4.82:  NSPS  is  better/worse  than  the  previous  system  for  Pay  Levels. 

Figures  4.83  and  4.84:  NSPS  is  better/worse  than  the  previous  system  for  Performance 
Management. 
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NS  PS  Spiral  1  Evaluation  Report 


Figures  4.85  and  4.86:  NSPS  is  better/worse  than  the  previous  system  for  Labor/Management 
Relations. 

Figures  4.87  and  4.88:  NSPS  is  better/worse  than  the  previous  system  for  Recognition  and 
Rewards. 

Figures  4.89  and  4.90:  NSPS  is  better/worse  than  the  previous  system  for  Workforce  Shaping. 

Figures  4.91  and  4.92:  NSPS  is  better/worse  than  the  previous  system  for  Conduct  and 
Discipline. 

Figures  4.93  through  4.95  (May  2007):  -  Better/Worse:  Compared  to  the  previous  personnel 
system,  NSPS  is... 

Figure  4.96:  Agree/Disagree  [Supervisors  of  NSPS  employees]:  The  time  it  takes  to  develop  good 
performance  plans,  discuss  and  assess  performance,  and  give  feedback  is  worthwhile. 

Figure  4.97:  Agree/Disagree  [Supervisors  of  NSPS  employees]:  My  employees'  performance 
plans  included  stretch  requirements  as  well  as  ones  that  could  more  easily  be  achieved. 

Figure  4.98:  Agree/Disagree:  The  use  of  Contributing  Factors  to  appraise  my  performance 
affects  my  approach  to  accomplishing  my  job  objectives. 

Figure  4.99:  Agree/Disagree:  The  five-level  NSPS  performance  rating  system  provides 

meaningful  performance  differentiation  among  employees. 

Figure  4.100:  Agree/Disagree:  The  Pay  Pool  Panel  helps  ensure  that  the  performance  rating  and 
payout  process  is  equitable  in  my  organization. 

Figure  4.101:  How  useful  was  the  information  provided  about  rating  outcomes  in  your  Pay 
Pool? 

Figure  4.102:  Agree/Disagree  [Supervisors  of  NSPS  employees]:  I  effectively  answered 
subordinates'  questions  about  NSPS  performance  rating,  shares,  and  payout  distribution. 

Figure  4.103:  Agree/Disagree  [Supervisors  of  NSPS  employees]:  The  rating  assessment  input  I 
provided  for  my  employee(s)  was  effective  in  supporting  the  rating  I  recommended. 

Figure  4.104:  Agree/Disagree  [Supervisors  of  NSPS  employees]:  As  implemented,  pay  for 
performance  at  my  organization  was  a  fair  reflection  of  my  ratees1  performance. 

Figures  4.105  through  4.107:  Agree/Disagree  [Supervisors  of  NSPS  employees]:  I  have  the  tools, 
training,  and  information  I  need  to  make  pay  decisions  under  NSPS. 

Figure  4.108:  How  useful  was  the  NSPS  training  you  received  on...  [Includes  only  "useful/very 
useful  responses"]. 

Figure  4.109:  [Employees  not  under  NSPS]:  Which  of  the  following  is  the  most  important  NSPS 
training  you  would  like  to  see  provided? 

Figure  4.110:  [Employees  already  under  NSPS]:  Which  of  the  following  is  the  most  important 
additional  NSPS  training  you  would  like  to  see  provided? 

Figure  4.111:  [Employees  not  under  NSPS]:  Which  of  the  following  is  the  most  important 
supervisor  skill  under  NSPS? 

Figure  4.112:  [Employees  already  under  NSPS]:  Which  of  the  following  is  the  most  important 
supervisor  skill  under  NSPS? 
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Figures  4.113  through  4.115:  Overall,  what  type  of  effect  do  you  think  NSPS  will  have  on 
personnel  practices  in  the  DoD? 

Summary: 

Spiral  1.1  employees'  favorable  response  rates  increased  in  February  2008  for  all  of  the  "NSPS 
has  improved"  items  except  for  "ensuring  individual  performance  supports  mission 
effectiveness"  (Figures  4.76).  The  favorable  response  rates  for  Spiral  1.3  increased  on  all  of 
these  items  except  for  "hiring  of  new  employees"  (Figures  4.58).  However,  the  unfavorable 
response  rates  for  Spirals  1.2  and  1.3  also  increased  on  all  of  the  "NSPS  has  improved"  items 
(Figures  4.59,  4.62,  4.65,  4.68,  4.71,  4.74,  4.77).  Spiral  1.1's  unfavorable  response  rates 
decreased  on  five  out  of  the  seven  "NSPS  has  improved"  items  (Figures  4.62,  4.65,  4.68,  4.71, 
4.74),  and  in  February  2008  the  favorable  response  rates  of  Spirals  1.1  and  1.2  were  higher  than 
the  unfavorable  response  rates  for  "ensuring  individual  performance  supports  mission 
effectiveness"  and  "improving  communication  between  supervisors  and  employees"  (Figures 
4.73,  4.74,  4.76,  and  4.77). 

There  was  a  slight  increase  between  May  2007  and  February  2008  in  the  portion  of 
respondents  who  indicated  that  NSPS  was  better  than  their  previous  personnel  system  for  pay 
levels,  performance  management,  and  recognition  and  rewards.  While  the  portion  of  Spiral  1.1 
employees  who  believed  that  NSPS  was  worse  than  the  previous  system  declined  in  February 
2008,  all  Spiral  1  populations  continued  to  have  employees  who  believed  NSPS  was  worse  than 
the  previous  system  than  those  who  believed  NSPS  was  better. 

There  was  consistently  more  agreement  than  disagreement  among  spirals  that  writing  NSPS 
performance  plans  and  conducting  feedback  sessions  with  employees  under  NSPS  is  worthwhile 
(Figures  4.96).  Spiral  1  supervisors  also  consistently  agreed  more  than  disagreed  that  their 
employees'  performance  plans  included  both  "stretch"  and  "more  easily  attainable"  job  objectives 
(Figures  4.97). 

Spiral  1  employees'  views  of  the  NSPS  performance  management  system  were  somewhat  mixed. 
Slightly  more  Spiral  1  employees  agreed  than  disagreed  that  the  use  of  contributing  factors  affected 
their  approach  to  accomplishing  their  job  objectives  (Figures  4.98).  However,  less  than  30%  agreed 
that  the  five-level  ratings  system  in  NSPS  meaningfully  distinguishes  levels  of  performance, 
compared  to  about  40%  who  disagreed  (Figures  4.99).  Less  than  30%  agreed  that  the  Pay  Pool 
Panel  helps  ensure  equitable  ratings  and  payouts,  while  about  40%  disagreed  (Figures  4.100). 
However,  over  40%  of  Spiral  1  employees  said  that  the  information  provided  about  ratings 
outcomes  was  useful,  compared  with  about  25%  who  said  this  information  was  not  useful  (Figures 
4.101). 

Spiral  1.1  supervisors  appeared  to  be  the  most  confident  of  the  Spiral  1  supervisors  in  their  abilities 
to  effectively  answer  their  subordinates'  questions  regarding  NSPS  (Figures  4.102).  Similarly,  Spiral 
1.1  supervisors,  who  had  been  through  two  ratings  cycles  as  of  February  2008,  were  the  most 
confident  that  their  ratings  assessment  input  supported  their  recommended  performance  ratings 
(Figures  4.103).  Supervisors  in  Spiral  1.1  were  much  more  likely  than  those  in  Spiral  1.2  or  1.3  to 
agree  that  pay  for  performance  was  a  fair  reflection  of  their  employees'  performance  (Figures 
4.104).  However,  a  higher  portion  of  Spiral  1.1  supervisors  than  Spiral  1.2  and  1.3  supervisors 
disagreed  with  this  item. 

All  Spiral  1  supervisors  had  higher  favorable  responses  than  Spiral  2  supervisors  to  "I  have  the  tools, 
training,  and  information  I  need  to  make  pay  decisions  under  NSPS"  (Figures  4.105  through  4.107). 
Spiral  1.2  supervisors  showed  the  largest  increase  in  favorable  responses  between  2007  and  2008. 
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Generally,  all  spirals  have  positive  and  stable  responses  regarding  the  usefulness  of  NSPS  training 
courses  (Figures  4.108).  Employees  -  both  under  and  not  under  NSPS  -  would  most  like  to  see  more 
training  on  the  Performance  Appraisal  System  (Figures  4.109  and  4.110).  Employees  already  under 
NSPS  appear  to  be  more  concerned  with  learning  more  about  the  Pay  Pool  Panel  Process  than 
employees  not  yet  under  NSPS. 

Employees  -  both  under  and  not  under  NSPS  -  believe  that  "communicating  performance 
expectations  and  making  fair  personnel  decisions"  are  the  two  most  important  supervisor  skills 
under  NSPS  (Figures  4.111  and  4.112).  Employees  already  under  NSPS  appear  to  be  less  concerned 
with  supervisors  "making  fair  personnel  decisions"  (compared  to  those  not  under  NSPS)  and  more 
concerned  with  supervisors  "communicating  performance  expectations." 

Respondents  who  either  are  under  NSPS  or  supervise  NSPS  personnel  have  more  defined  views  of 
NSPS'  effects  -  both  positive  and  negative  -  on  the  DoD  (Figures  4.113  through  4.115).  Opinions  on 
NSPS'  overall  impact  grew  decidedly  more  negative  in  Spiral  1.1  from  May  2006  to  November  2006 
but  then  leveled  off  in  May  2007. 

4.6.1.  Potential  vs.  Already-Observed  Effects  of  NSPS 


Non-NSPS  Employees:  Do  you  agree  that  NSPS  WILL 
improve  hiring  new  employees? 


Total  Non-NSPS  Agree 
X  Total  Non-NSPS  Disagree 

Figures  4.57-  NSPS  Will  Improve  Hiring 


Non-NSPS  employees  continued  to  disagree  more  than  agree  that  NSPS  will  improve  hiring. 
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NSPS  Employees:  Do  you  agree  that  NSPS  HAS 
improved  hiring  new  employees?  -  Agree 


Spiral  1.1— ♦—Spiral  1.2-1— Spiral  1.3 


NSPS  Employees:  Do  you  agree  that  NSPS  HAS 
improved  hiring  new  employees?  -  Disagree 
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Figures  4.58  and  4.59  -  NSPS  Has  Improved  Hiring 

Unfavorable  response  rates  remained  higher  than  favorable  response  rates  for  all  three  Spiral  1 
populations  in  February  2008.  Spiral  l.l's  unfavorable  response  rate  rose  from  May  2007  to  February 
2008,  but  by  much  less  than  it  did  between  May  2006  and  May  2007. 


Non-NSPS  Employees:  Do  you  agree  that  NSPS  WILL 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  Non-NSPS  Agree 
X  Total  Non-NSPS  Disagree 

Figures  4.60  -  NSPS  Will  Improve  Correcting  Poor  Performers 

Among  non-NSPS  employees  there  was  a  slight  increase  in  unfavorable  responses  in  February  2008  and 
leveling  off  from  a  steady  decline  in  favorable  responses  from  May  2006  to  May  2007.  Non-NSPS 
employees'  favorable  response  rate  was  still  slightly  higher  than  their  unfavorable  response  rate  in 
February  2008. 
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NSPS  Employees:  Do  you  agree  that  NSPS  HAS 
improved  disciplining/correcting  poor  work 


NSPS  Employees:  Do  you  agree  that  NSPS  HAS 
improved  disciplining/correcting  poor  work 


■Spiral  1.1— ♦— Spiral  1.2-B- Spiral  1.3 


■Spiral  1.1 


■Spiral  1.2 


Spiral  1.3 


Figures  4.61  and  4.62  -  NSPS  Has  Improved  Correcting  Poor  Performers 

Favorable  response  rates  increased  slightly  from  May  2007  to  February  2008.  Spiral  1.1's  unfavorable 
response  rate  declined  slightly  between  May  2007  and  February  2008,  while  the  unfavorable  response 
rates  for  Spirals  1.2  and  1.3  increased  substantially. 


Non-NSPS  Employees:  Do  you  agree  that  NSPS  WILL 
improve  rewarding  good  work  performance? 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  Non-NSPS  Agree 
X  Total  Non-NSPS  Disagree 

Figures  4.63-  NSPS  Will  Improve  Rewarding  Good  Performance 

Non-NSPS  employees  have  steadily  become  less  positive  in  their  answers  that  NSPS  will  reward  good 
work  performance,  although  the  favorable  response  rate  is  still  higher  than  the  unfavorable  response 
rate.  The  unfavorable  response  rate  has  increased  slightly  from  November  2006  to  February  2008. 
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NSPS  Employees:  Do  you  agree  that  NSPS  HAS 
improved  rewarding  good  work  performance?  - 


Spiral  1.1— ♦—Spiral  1.2-B— Spiral  1.3 


NSPS  Employees:  Do  you  agree  that  NSPS  HAS 
improved  rewarding  good  work  performance?  - 
Disagree 
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Figures  4.64  and  4.65  -  NSPS  Has  Improved  Rewarding  Good  Performance 

All  NSPS  Spiral  1  populations  became  more  favorable  that  NSPS  has  improved  rewarding  good 
performance  in  February  2008,  after  the  second  performance  payout  for  Spiral  1.1  and  the  first  for 
Spirals  1.2  and  1.3.  Unfavorable  responses  for  Spiral  1.1  declined  in  February  2008,  although  they  rose 
substantially  for  Spirals  1.2  and  1.3.  Unfavorable  responses  are  higher  than  favorable  responses  for  all 
three  groups. 


Non-NSPS  Employees:  Do  you  agree  that  NSPS  WILL 
improve  linking  pay  to  performance? 
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Figures  4.66-  NSPS  Will  Improve  Linking  Pay  to  Performance 

Non-NSPS  employees  have  steadily  become  less  positive  in  their  answers  that  NSPS  will  improve  linking 
pay  to  performance  with  their  favorable  responses  approximately  even  with  their  non-favorable 
responses  in  February  2008. 
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NSPS  Employees:  Do  you  agree  that  NSPS  HAS  NSPS  Employees:  Do  you  agree  that  NSPS  HAS 

improved  linking  pay  to  performance?  -  Agree  improved  linking  pay  to  performance?  -  Disagree 


—♦—Spiral  1.1— ♦—Spiral  1.2 — ■ — Spiral  1.3  — ♦—  Spiral  1.1  —♦—Spiral  1.2  — ■ —  Spiral  1.3 

Figures  4.67  and  4.68  -  NSPS  Has  Improved  Linking  Pay  to  Performance 

All  three  Spiral  1  populations  increased  their  favorable  response  rates  to  "NSPS  has  improved  linking  pay 
to  performance"  in  February  2008,  after  the  first  performance  payouts  for  Spirals  1.2  and  1.3.  Spiral 
1.1's  unfavorable  response  rate  also  declined  after  its  second  payout,  while  the  unfavorable  response 
rates  for  Spirals  1.2  and  1.3  increased  significantly. 


Non-NSPS  Employees:  Do  you  agree  that  NSPS  WILL 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  Non-NSPS  Agree 
X  Total  Non-NSPS  Disagree 

Figures  4.69-  NSPS  Will  Improve  Classification  of  Jobs 

Non-NSPS  employees'  favorable  responses  that  NSPS  will  improve  classification  of  jobs  by  pay  grade/pay 
band  have  steadily  declined.  The  non-NSPS  employee  population  disagreed  slightly  more  than  it  agreed 
that  NSPS  would  improve  classification  of  jobs. 
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NSPS  Employees:  Do  you  agree  that  NSPS  HAS  NSPS  Employees:  Do  you  agree  that  NSPS  HAS 

improved  classification  of  jobs  by  pay  improved  classification  of  jobs  by  pay 


—♦—Spiral  1.1— ♦—Spiral  1.2— ■ — Spiral  1.3  —♦—Spiral  1.1  —♦—Spiral  1.2  — ■ —  Spiral  1.3 

Figures  4.70  and  4.71  -  NSPS  Has  Improved  Classification  of  Jobs 

Spiral  1.1's  unfavorable  response  rate  declined  slightly  in  February  2008,  although  the  unfavorable 
response  rates  for  all  of  Spiral  1  were  still  higher  than  their  respective  favorable  response  rates. 


Non-NSPS  Employees:  Do  you  agree  that  NSPS  WILL 
improve  communication  between  supervisors  and 
employees? 
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Figures  4.72-  NSPS  Will  Improve  Communication 

The  non-NSPS  employees'  favorable  response  rate  was  approximately  equal  to  their  unfavorable 
response  rate  on  this  question. 
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NSPS  Employees:  Do  you  agree  that  NSPS  HAS  NSPS  Employees:  Do  you  agree  that  NSPS  HAS 

improved  communication  between  supervisors  and  improved  communication  between  supervisors  and 


■Spiral  1.1— ♦—  Spiral  1.2-1- Spiral  1.3 


•Spiral  1.1 


■Spiral  1.2 


■Spiral  1.3 


Figures  4.73  and  4.74  -  NSPS  Has  Improved  Communication 

Spiral  1.1's  favorable  response  rate  was  slightly  higher  than  its  unfavorable  response  rate  in  February 
2008.  The  unfavorable  response  rates  for  Spirals  1.2  and  1.3  increased  significantly  between  May  2007 
and  February  2008. 


Non-NSPS  Employees:  Do  you  agree  that  NSPS  WILL 
improve  ensuring  individual  performance  supports 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 

Total  Non-NSPS  Agree 
X  Total  Non-NSPS  Disagree 

Figures  4.75-  NSPS  Will  Improve  Linking  Performance  to  Mission 

A  slightly  higher  portion  of  the  non-NSPS  employee  population  agreed  that  NSPS  will  improve  this 
measure  in  February  2008. 
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NSPS  Employees:  Do  you  agree  that  NSPS  HAS 
improved  ensuring  individual  performance  supports 
organizational  mission  effectiveness?  -  Agree 


NSPS  Employees:  Do  you  agree  that  NSPS  HAS 
improved  ensuring  individual  performance  supports 
organizational  mission  effectiveness?  -  Disagree 


-Spiral  1.1— ♦—  Spiral  1.2— ■ —  Spiral  1.3 


-Spiral  1.1 


-Spiral  1.2 


-Spiral  1.3 


Figures  4.76  and  4.77  -  NSPS  Has  Improved  Linking  Performance  to  Mission 

Spirals  1.1  and  1.3  still  had  a  slightly  higher  portion  of  respondents  agree  than  disagree  with  this  item  in 
February  2008.  Spiral  1.2's  unfavorable  response  rate  increased  significantly  from  May  2007  and  was 
greater  than  its  favorable  response  rate  in  February  2008. 


4.6.2.  Comparisons  of  NSPS  to  the  Previous  Personnel  System3 


For  employees  in  your  organization,  NSPS  is  BETTER/MUCH  BETTER 
than  your  previous  personnel  system  for ... 
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Figures  4.78-  NSPS  Is  Better  than  Previous  System  - 1 


Hiring, 
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and 

promotion 


There  were  increases  between  May  2007  and  February  2008  in  the  portion  of  respondents  who 
indicated  that  NSPS  was  better  than  their  previous  personnel  system  for  pay  levels  and  recognition  and 
rewards. 


This  series  of  questions  was  asked  for  the  first  time  in  May  2007.  It  is  directed  to  respondents  who  identify  themselves  as 
either  already  under  NSPS  or  responsible  for  supervising  employees  under  NSPS  as  of  the  date  of  the  survey. 
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How  does  NSPS  compare  with  your  previous  personnel 
system  for  hiring,  placement,  and  promotions?  - 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07  Feb-08 
♦  Spiral  1.1  ♦  Spiral  1.2— ■— Spiral  1.3 


How  does  NSPS  compare  with  your  previous  personnel 
system  for  hiring,  placement,  and  promotions?  - 
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♦  Spiral  1.1  ♦  Spiral  1.2— ■—  Spiral  1.3 


Figures  4.79  and  4.80  -  NSPS  Is  Better/Worse  for  Hiring,  Placement,  and  Promotion 

Spiral  1.1's  unfavorable  response  rate  concerning  the  relative  effects  of  NSPS  on  hiring,  placement,  and 
promotion  was  consistent  between  May  2007  and  February  2008,  although  the  unfavorable  response 
rates  for  Spirals  1.2  and  1.3  increased  significantly  during  this  period. 


How  does  NSPS  compare  with  your  previous 
personnel  system  for  pay  levels?  - 


Spiral  1.1— ♦—Spiral  1.2— ■ —  Spiral  1.3 


How  does  NSPS  compare  with  your  previous 
personnel  system  for  pay  levels?  - 


-Spiral  1.1— ♦—Spiral  1.2— ■ —  Spiral  1.3 


Figures  4.81  and  4.82  -  NSPS  Is  Better/Worse  for  Pay  Levels 

Both  favorable  and  unfavorable  response  rates  for  Spirals  1.2  and  1.3  rose  between  May  2007  and 
February  2008  concerning  the  relative  effect  of  NSPS  on  pay  levels.  Spiral  1.1's  favorable  response  rate 
also  rose  while  its  unfavorable  response  rate  declined  slightly,  although  it  is  still  much  higher  than  the 
favorable  response  rate. 
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How  does  NSPS  compare  with  your  previous  How  does  NSPS  compare  with  your  previous 

personnel  system  for  performance  management?  -  personnel  system  for  performance  management?  - 


-♦—Spiral  1.1— ♦—  Spiral  1.2-1- Spiral  1.3  — Spiral  1.1— ♦—  Spiral  1.2-1- Spiral  1.3 

Figures  4.83  and  4.84  -  NSPS  Is  Better/Worse  for  Performance  Management 

On  the  relative  effect  of  NSPS  on  performance  management,  the  favorable  response  rates  of  all  three 
Spiral  1  populations  increased  between  May  2007  and  February  2008.  Spiral  1.1's  unfavorable  response 
rate  declined  slightly  while  the  unfavorable  response  rates  for  Spirals  1.2  and  1.3  increased  significantly. 


How  does  NSPS  compare  with  your  previous  How  does  NSPS  compare  with  your  previous 

personnel  system  for  labor/management  relations?  ■  personnel  system  for  labor/management  relations?  ■ 


-♦—Spiral  1.1— ♦—  Spiral  1.2-1- Spiral  1.3  -♦—Spiral  1.1— ♦—  Spiral  1.2-1- Spiral  1.3 

Figures  4.85  and  4.86  -  NSPS  Is  Better/Worse  for  Labor-Management  Relations 

All  three  Spiral  1  populations  maintained  very  low  favorable  response  rates  in  February  2008  concerning 
the  relative  effect  of  NSPS  on  labor-management  relations.  Spiral  1.1's  unfavorable  response  rate  was 
constant  between  May  2007  and  February  2008. 
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How  does  NSPS  compare  with  your  previous  How  does  NSPS  compare  with  your  previous 
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Figures  4.87  and  4.88  -  NSPS  Is  Better/Worse  for  Recognition  and  Rewards 

Both  favorable  and  unfavorable  response  rates  concerning  NSPS'  relative  effect  on  recognition  and 
rewards  rose  in  February  2008,  with  the  exception  of  the  decline  in  Spiral  1.1's  unfavorable  response 
rate. 


personnel  system  for  recognition  and  rewards?  - 


personnel  system  for  recognition  and  rewards?  - 


■Spiral  1.1-#- Spiral  1.2— ■—Spiral  1.3 


■Spiral  1.1— ♦— Spiral  1.2-1- Spiral  1.3 


How  does  NSPS  compare  with  your  previous  How  does  NSPS  compare  with  your  previous 

personnel  system  for  workforce  shaping?  -  personnel  system  for  workforce  shaping?  - 


-♦-Spiral  1.1-#- Spiral  1.2-1- Spiral  1.3  -♦-Spiral  i.i-*- Spiral  1.2-1- Spiral  1.3 

Figures  4.89  and  4.90  -  NSPS  Is  Better/Worse  for  Workforce  Shaping 

Spiral  1.1's  favorable  and  unfavorable  response  rates  on  workforce  shaping  were  roughly  consistent 
between  May  2007  and  February  2008,  while  Spiral  1.2  and  1.3's  unfavorable  response  rates  increased 
significantly. 


4-32 


May  IS,  2009 


NSPS  Spiral  1  Evaluation  Report 


4.  Analysis  of  SO FS-C  Data 


How  does  NSPS  compare  with  your  previous  How  does  NSPS  compare  with  your  previous 

personnel  system  for  conduct  and  discipline?  -  personnel  system  for  conduct  and  discipline? 


-♦—Spiral  1.1-4—  Spiral  1.2-B— Spiral  1.3  — Spiral  1.1-4—  Spiral  1.2-B— Spiral  1.3 

Figures  4.91  and  4.92  -  NSPS  Is  Better/Worse  for  Conduct  and  Discipline 

Spiral  1.1's  favorable  and  unfavorable  response  rates  on  conduct  and  discipline  were  roughly  consistent 
between  May  2007  and  February  2008,  while  Spiral  1.2's  and  1.3's  unfavorable  response  rates  increased 
significantly. 


Compared  to  the  Previous  Personnel  System, 
NSPS  is...  -  May  2007 


Spiral  1.1  Spiral  1.2  Spiral  1.3 

■  Better/Much  Better  □  Neither  Better  nor  Worse  ■  Worse/Much  Worse 


Figure  4.93-  NSPS  Is  Better  than  Previous  System  -  2 
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Compared  to  your  previous  personnel  system,  NSPS 
is ...  "Better/Much  Better" 
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Compared  to  your  previous  personnel  system,  NSPS 
is ...  "Worse/Much  Worse" 


Oct-04  Apr-05  Nov-05  May-06  Nov-06  May-07 
—♦—Spiral  1.1  ♦  Spiral  1.2  M  Spiral  1.3 


Figures  4.94  and  4.95  -  How  NSPS  Compares  to  Your  Previous  System 

A  small  portion  of  respondents  believes  that  NSPS  is  better  than  their  previous  personnel  system  overall. 
Attitudes  became  less  neutral  and  more  negative  for  Spirals  1.1  and  1.2  between  November  2006  and 
May  2007. 

4.6.3.  NSPS  Performance  Plans  and  the  Pay  Pool  Process4 


Supervisors:  The  time  it  takes  to  develop  good  performance  plans,  discuss 
and  assess  performance,  and  give  feedback  is  worthwhile  -  February  2008 
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Spiral  1.1  Spiral  1.2  Spiral  1.3 

■  Agree/Strongly  Agree  □  Neither  Agree  nor  Disagree  ■  Disagree/Strongly  Disagree 

Figure  4.96  -  Time  Is  Worthwhile 

There  is  consistent  agreement  among  Spiral  1  supervisors  that  writing  NSPS  performance  plans  and 
conducting  feedback  sessions  with  employees  under  NSPS  is  worthwhile. 


4  These  two  items  were  asked  for  the  first  time  in  February  2008.  They  were  directed  to  supervisors  who  were  under  NSPS  or 
supervised  NSPS  employees  as  of  February  2008. 
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Supervisors:  My  employees'  performance  plans  included  stretch 
requirements  as  well  as  ones  that  could  more  easily  be  achieved  - 
February  2008 
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There  is  relatively  consistent  agreement  among  Spiral  1  supervisors  that  NSPS  employees'  performance 
plans  include  both  "stretch"  and  "easier  to  achieve"  objectives. 


Spiral  1.1  Spiral  1.2  Spiral  1.3 


Agree/Strongly  Agree  □  Neither  Agree  nor  Disagree  ■  Disagree/Strongly  Disagree 

Figure  4.97-  Assessment  of  Performance  Plans 


The  use  of  Contributing  Factors  to  appraise  my  performance  affects  my 
approach  to  accomplishing  my  job  objectives  -  February  2008 
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■  Agree/Strongly  Agree  □  Neither  Agree  nor  Disagree  ■  Disagree/Strongly  Disagree 

Figure  4.98-  Effects  of  Contributing  Factors  on  Performance 

More  employees  in  Spiral  1  agree  than  disagree  that  the  use  of  contributing  factors  in  their  performance 
plans  influences  the  way  in  which  they  attempt  to  accomplish  their  job  objectives. 
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The  five-level  NSPS  performance  rating  system  provides  meaningful 
performance  differentiation  among  employees 
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D  Agree/Strongly  Agree  □  Neither  Agree  nor  Disagree  ■  Disagree/Strongly  Disagree 

Figure  4.99-  Meaningful  Differentiation  in  Performance 

A  relatively  low  portion  of  NSPS  employees  agrees  that  NSPS  allows  for  meaningful  distinction  in  levels 
of  performance. 


The  Pay  Pool  Panel  helps  ensure  that  the  performance  rating 
and  payout  process  is  equitable  in  my  organization 
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Spiral  1.1  Spiral  1.2  Spiral  1.3 

□  Agree/Strongly  Agree  □  Neither  Agree  nor  Disagree  ■  Disagree/Strongly  Disagree 

Figure  4.100-  Efficacy  of  the  Pay  Pool  Panel  Process 

More  employees  in  Spiral  1  disagree  than  agree  that  the  pay  pool  panel  helps  ensure  equitable  ratings 
and  performance  payouts. 
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How  useful  was  the  information  provided  about  rating  outcomes  in  your 
Pay  Pool?  -  February  2008 


Spiral  1.1  Spiral  1.2  Spiral  1.3 


■  Useful/Very  Useful  □  Neither  Useful  nor  Useless  ■  Useless/Very  Useless 


Figure  4.101-  Usefulness  of  Pay  Pool  Information 

More  than  one-third  of  NSPS  employees  believed  that  information  on  ratings  outcomes  from  the  pay 
pool  was  useful,  while  26%-30%  thought  it  was  not  useful. 


Supervisors:  I  effectively  answered  my  subordinates'  questions  about 
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Spiral  1.1  Spiral  1.2  Spiral  1.3 

■  Agree/Strongly  Agree  □  Neither  Agree  nor  Disagree 

■  Disagree/Strongly  Disagree  □  Not  Applicable 

Figure  4.102  - 1  Effectively  Answered  Questions 


their  NSPS  performance  rating,  shares,  and  payout  distribution  - 


February  2008 


Supervisors  in  Spiral  1.1  were  more  confident  than  those  in  Spirals  1.2  or  1.3  in  their  abilities  to  answer 
employees'  questions  concerning  NSPS. 
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Supervisors:  The  rating  assessment  input  I  provided  for  my  employee(s) 
was  effective  in  supporting  the  rating  I  recommended  -  February  2008 
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Spiral  1.1  Spiral  1.2  Spiral  1.3 

■  Agree/Strongly  Agree  □  Neither  Agree  nor  Disagree 

■  Disagree/Strongly  Disagree  □  Not  Applicable 

Figure  4.103- 1  Effectively  Supported  Ratings  Recommendations 

Supervisors  in  Spiral  1  were  more  confident  than  those  in  Spirals  1.2  or  1.3  in  their  abilities  to  fully 
support  their  ratings  recommendations. 


Supervisors:  As  implemented,  pay  for  performance  at  my  organization  was 
a  fair  reflection  of  my  ratees'  performance  -  February  2008 
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■  Agree/Strongly  Agree  □  Neither  Agree  nor  Disagree 

■  Disagree/Strongly  Disagree  □  Not  Applicable 

Figure  4.104-  Pay  for  Performance  Was  Fair  to  My  Employees 

Supervisors  in  Spiral  1.1  were  much  more  likely  than  those  in  Spirals  1.2  or  1.3  to  agree  that  pay  for 
performance  was  a  fair  reflection  of  their  employees'  performance. 
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4.6.4.  NSPS  Communications  and  Training 


Supervisors:  I  AGREE/STRONGLY  AGREE  that  I  have  the  tools,  training,  and 
information  I  need  to  make  pay  decisions  under  NSPS 


Spiral  1.1  Spiral  1.2  Spiral  1.3 

■  May-07  B  Feb-08 


Figure  4.105- 1  Can  Make  Pay  Decisions 


Supervisors:  Do  you  agree  that  you  have  the  tools, 
training,  and  information  you  need  to  make  pay 
decisions  under  NSPS?  -  Agree 


Supervisors:  Do  you  agree  that  you  have  the  tools, 
training,  and  information  you  need  to  make  pay 
decisions  under  NSPS?  -  Disagree 
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—♦—Spiral  1.1  —♦—Spiral  1.2  Spiral  1.3  —♦—Spiral  1.1  —♦—Spiral  1.2  -B—  Spiral  1.3 

Figures  4.106  and  4.107  -  Can  You  Make  Pay  Decisions 

Supervisors  in  all  NSPS  spirals  are  generally  more  positive  than  negative  about  their  abilities  to  make  pay 
decisions  under  NSPS. 
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NSPS  Communications  and  Training  (continued5) 


NSPS  Training  that  I  Received  was  USEFUL/VERY  USEFUL  on... 


NSPS  101  Web-  HRE  for  Performance  Performance  Pay  Pool 
Based  Managers,  Management  Management  Management 

Orientation  Supervisors,  for  Managers  for  Employees 

and  Employees  and  Supervisors 

■  May-07  ■  Feb-08 


SMART 

Objectives/ 

iSuccess 


Figure  4.108-  NSPS  Training  Was  Useful 

Generally,  there  are  positive,  stable  responses  regarding  the  usefulness  of  NSPS  training  courses. 


Most  Desired  NSPS  Training  -  Employees  NOT  under  NSPS 

■  May-07  □  Feb-08 
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Use  of  Pay-Setting  Hiring,  Placement,  PayPool  Panel  Performance  Other 

Flexibility  a nd  Adva ncement  Process  Appraisal  System 

Figure  4.109-  Most  Desired  Training  - 1 


This  set  of  questions  on  the  usefulness  of  NSPS  training  was  asked  of  respondents  who  indicated  that  they  had  received 
training  on  the  given  subjects. 
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Most  ADDITIONAL  Desired  NSPS  Training  -  Employees  ALREADY  under  NSPS 

a  Mav-07  □  Fcb- 
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Use  of  Pay-Setting  Hiring,  Placement,  Pay  Pool  Panel  Performance  Other 

Flexibility  and  Advancement  Process  Appraisal  System 

Figure  4.110-  Most  Desired  Training  -  2 

Employees  in  both  groups  (NSPS  and  non-NSPS)  would  most  like  to  see  more  training  on  the 
Performance  Appraisal  System.  Employees  already  under  NSPS  appear  to  be  more  concerned  with 
learning  more  about  the  pay  pool  panel  process  than  employees  not  yet  under  NSPS. 


Most  Important  Supervisor  Skill  under  NSPS  -  Employees  NOT  under  NSPS 

■  May-07  □  Feb-08 
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Figure  4.111-  Most  Important  Skill  - 1 
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Most  Important  Supervisor  Skill  under  NSPS  -  Employees  ALREADY  under  NSPS 


Communicating  Career  Dealing  with  TeachingJob  Motivating  Communicating  Making  Fair  Encouraging  Other 

Performance  Counseling  Poor  Skills  Employees  to  Effectively  with  Personnel  Teamwork  and 

Expectations  Performers  Perform  Well  Diverse...  Decisions  Cooperation 


Figure  4.112-  Most  Important  Skill  -  2 


Employees  in  both  groups  (NSPS  and  non-NSPS)  believe  that  communicating  performance  expectations 
and  making  fair  personnel  decisions  are  the  two  most  important  supervisor  skills  under  NSPS. 
Employees  already  under  NSPS  appear  to  be  less  concerned  with  supervisors  making  fair  personnel 
decisions  and  more  concerned  with  supervisors  communicating  performance  expectations. 


4. 6.5.  Overall  Impact  of  NSPS 


Overall,  what  type  of  impact  do  you  think  NSPS  will  have  on  personnel 
practices  in  the  DoD?  -  May  2007 
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Figure  4.113-  Overall  Impact  of  NSPS 
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Overall,  what  type  of  impact  will  NSPS  have  on 
personnel  practices  in  the  DoD?  -  Positive 


Overall,  what  type  of  impact  will  NSPS  have  on 
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Figures  4.114  and  4.115  -  What  Impact  Will  NSPS  Have 

Respondents  who  have  some  first-hand  experience  with  NSPS  (either  under  NSPS  or  supervise  NSPS 
personnel)  have  more  defined  views  of  NSPS'  effects— both  positive  and  negative— on  the  DoD. 
Opinions  on  NSPS'  overall  impact  grew  decidedly  more  negative  in  Spiral  1.1  from  May  2006  to 
November  2006,  but  in  May  2007  Spiral  1.1's  favorable  response  level  slowed  its  decrease  as  its 
unfavorable  response  rate  continued  to  increase. 
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5.  Integrated  Findings 


Chapter  5  integrates  the  study's  findings  across  data  sources.  For  each  KPP  component,  we 
examine  findings  from  analyses  of  DCPDS  personnel  data  (Chapter  2),  focus  groups  and 
interviews  conducted  on  12  site  visits  (Chapter  3),  and  responses  to  SOFS-C  survey  questions 
(Chapter  4).  A  summary  integrates  the  relevant  findings  for  each  KPP  component. 

5.1.  KPP  1:  High  Performing  Workforce  and  Management 

5.1.1.  Stronger  Link  between  Total  Compensation  and  Performance 

DCPDS 

■  Analyses  show  a  positive  link  between  performance  ratings,  pay  increases,  and  bonuses. 
The  analyses  also  show  that  performance  ratings  rise  with  pay  band. 

SOFS-C 

■  NSPS  is  better/worse  than  the  previous  system  for  Pay  Levels.  There  was  a  slight 
increase  between  May  2007  and  February  2008  in  the  portion  of  respondents  who 
indicated  that  NSPS  was  better  than  their  previous  personnel  system  for  Pay  Levels,  even 
as  unfavorable  response  levels  were  higher  than  favorable  levels. 

■  NSPS  is  better/worse  than  the  previous  system  for  Recognition  and  Rewards.  All  spirals 
show  strongly  higher  unfavorable  than  favorable  response  levels  that  NSPS  is 
better/worse  for  recognition  and  rewards  from  May  2007  to  February  2008,  while  all 
spirals  increased  strongly  in  favorable  responses  over  this  period. 

■  NSPS  has  improved  rewarding  good  work  performance  (Agree/Disagree).  Unfavorable 
responses  are  still  higher  than  favorable  responses  for  all  three  spirals  that  NSPS  has 
rewarded  good  work  performance,  but  all  spirals  show  a  sharp  increase  in  favorable 
responses  from  May  2007  to  February  2008. 

■  NSPS  has  improved  linking  pay  to  performance  (Agree/Disagree).  All  Spiral  1  populations 
increased  their  favorable  response  rates  to  "NSPS  has  improved  linking  pay  to 
performance"  in  February  2008.  All  three  spirals  have  higher  unfavorable  response  rates 
than  favorable  to  this  item.  Even  though  Spiral  1.1's  unfavorable  response  rate  declined 
after  its  second  payout,  the  unfavorable  response  rates  for  Spirals  1.2  and  1.3  increased 
significantly. 

■  Pay  raises  depend  on  how  well  employees  perform  their  jobs  (Agree/Disagree).  All  NSPS 
Spirals  are  substantially  more  favorable  and  less  unfavorable  than  the  non-NSPS 
population  that  their  pay  raises  depend  on  job  performance.  Both  NSPS  and  non-NSPS 
groups  have  increased  in  favorable  responses  since  November  2006. 

Focus  Groups  and  Interviews 

■  Not  all  employees  see  a  clear  link  between  performance  and  pay  under  NSPS.  Some 
employees  and  supervisors  believe  that  the  pay  pools  were  instructed  to  give  primarily 
ratings  of  3,  resulting  in  placing  some  marginal  performers  and  solid  performers  in  the 
same  rating  level  so  that  they  receive  similar  compensation. 

Summary: 

Findings  from  employee  focus  groups  and  senior  leader  interviews  generally  agree  with  SOFS-C 
survey  results  showing  employees  more  unsure  than  sure  that  NSPS  has  resulted  in 
improvements  relative  to  GS,  such  as  strengthening  the  link  between  pay  and  performance, 
rewarding  good  work  performance,  improving  pay  levels,  and  recognizing  and  rewarding 
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performance.  Analyses  of  DCPDS  data  show  a  clear  link  between  performance  ratings  and 

payouts. 

5.1.2.  Market-Sensitive  Salary  and  Wages 

SOFS-C 

■  Considering  everything,  how  satisfied  are  you  with  your  pay?  All  NSPS  groups  showed 
favorable  response  levels  far  above  unfavorable  levels. 

Focus  Groups  and  Interviews 

■  Many  employees  believe  that  salaries  are  more  market-sensitive  for  new  hires,  allowing 
organizations  to  recruit  employees  with  high-level  expertise  more  effectively.  However, 
because  salaries  are  not  market-sensitive  for  existing  employees,  some  perceive  inequity. 

Summary: 

While  current  employees  are  generally  satisfied  with  their  pay  under  NSPS,  some  perceive 

inequities  relative  to  new  hires  who  can  negotiate  higher  starting  salaries. 

5.1.3.  Performance  System  Provides  Ongoing  Feedback 

SOFS-C 

■  [Supervisors  of  NSPS  employees]:  The  time  it  takes  to  develop  good  performance  plans, 
discuss  and  assess  performance,  and  give  feedback  is  worthwhile  (Agree/Disagree).  All 

NSPS  groups  showed  higher  agreement  than  disagreement  that  the  time  it  takes  to 
develop  good  performance  plans,  discuss  and  assess  performance,  and  give  feedback  is 
worthwhile.  As  of  February  2008  there  is  consistently  higher  agreement  than 
disagreement  across  spiral  groups  that  discussions  with  one's  supervisors  about 
performance  are  worthwhile. 

■  Discussions  with  my  supervisor/team  leader  about  my  performance  are  worthwhile 
(Agree/Disagree).  As  of  February  2008  there  is  consistently  higher  agreement  than 
disagreement  across  spiral  groups  that  writing  NSPS  performance  plans  and  conducting 
feedback  sessions  with  employees  under  NSPS  are  worthwhile. 

■  The  five-level  NSPS  performance  rating  system  provides  meaningful  performance 
differentiation  among  employees  (Agree/Disagree).  A  relatively  low  portion  of  NSPS 
employees  agrees  that  NSPS  allows  for  meaningful  distinction  in  levels  of  performance. 

■  [Supervisors  of  NSPS  employees]:  My  employees'  performance  plans  included  stretch 
requirements  as  well  as  ones  that  could  more  easily  be  achieved  (Agree/Disagree). 

There  is  relatively  consistent  agreement  among  Spiral  1  supervisors  that  NSPS  employees' 
performance  plans  include  both  "stretch"  and  "easier  to  achieve"  objectives. 

■  The  use  of  Contributing  Factors  to  appraise  my  performance  affects  my  approach  to 
accomplishing  my  job  objectives  (Agree/Disagree).  Slightly  more  Spiral  1  employees 
agree  than  disagree  that  the  use  of  contributing  factors  in  their  performance  plans 
influences  the  way  in  which  they  attempt  to  accomplish  their  job  objectives. 

■  NSPS  has  improved  communication  between  supervisors  and  employees 
(Agree/Disagree).  All  spirals'  favorable  and  unfavorable  responses  are  fairly  equal  in 
agreeing  that  NSPS  has  improved  communication  between  supervisors  and  employees. 

■  [Employees  under  NSPS]:  Which  of  the  following  is  the  most  important  supervisor  skill 
under  NSPS?  In  February  2008  nearly  half  of  surveyed  NSPS  employees,  more  than  in 
2007,  were  most  concerned  with  supervisors'  communicating  performance  expectations 
effectively. 
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■  How  satisfied  are  you  with  the  recognition  you  receive  for  doing  a  good  job?  As  of 

February  2008,  all  NSPS  spirals  were  far  more  satisfied  with  the  recognition  they  receive 
for  doing  a  good  job  than  unsatisfied. 

Focus  Groups  and  Interviews 

■  There  were  varied  views  on  the  performance  evaluation  process.  While  some  supervisors 
and  employees  appreciated  that  NSPS  has  resulted  in  more  regular  and  formal 
performance  feedback,  others  indicated  that  the  interim  reviews  did  not  always  provide 
concrete  and  specific  suggestions  for  improvement  and  were  not  always  an  indication  of 
the  year-end  performance  rating.  Some  employees  viewed  self-assessments  as  an 
opportunity  to  showcase  contributions  to  their  organization's  mission,  while  others 
viewed  them  primarily  as  a  challenging  writing  assignment. 

Summary: 

Findings  from  employee  focus  groups  generally  agree  with  SOFS-C  results  that  regular  and 
formal  feedback  and  the  time  required  to  develop  performance  plans  are  worthwhile,  although 
SOFS-C  results  show  no  strong  indication  that  NSPS  has  improved  communications  between 
supervisors  and  employees.  While  SOFS-C  results  show  employees  are  satisfied  with  their 
performance  recognition,  they  also  indicate  that  meaningful  distinctions  in  performance  are 
not  readily  seen  under  NSPS.  This  view  aligns  somewhat  with  focus  group  findings  that  some 
employees  do  not  agree  that  ratings  match  performance. 

5.1.4.  Stronger  Link  between  Retention/Separation  and  Performance 

Focus  Groups  and  Interviews 

■  Many  employees,  and  not  just  those  who  received  a  rating  of  2,  are  concerned  about  the 
impact  of  NSPS  on  their  careers  and  their  opportunity  for  progression.  Employees  and 
supervisors  exhibited  a  strong  reluctance  to  accept  "valued  performer"— or  3— as  a  good 
rating.  A  majority  stated  they  cannot  articulate  what  they  must  do  to  exceed  a  3  rating.  A 
number  of  employees  expressed  concern  that  non-NSPS  organizations  have  negative 
perceptions  of  NSPS  3  ratings,  with  potential  impact  on  inter-system  transfers  (NSPS  to 
GS/APS)  or  transfers  to  other  NSPS  commands  with  perhaps  different  rating  standards. 
There  are  also  a  number  of  employees  who  lack  a  clear  understanding  of  career 
progression  under  NSPS. 

DCPDS 

■  In  2007  2.9%  of  those  in  NSPS  received  low  ratings  of  1  or  2  compared  with  .06%  of  non- 
NSPS.  The  2007  NSPS  and  non-NSPS  loss  rates  for  employees  with  low  ratings  differed 
greatly:  16.8%  for  NSPS  and  40.7%  for  non-NSPS.  The  overall  NSPS  loss  rate  in  2007  was 
6.5%. 

■  In  July  2008  1.7%  of  those  in  NSPS  received  low  ratings  of  1  or  2  compared  with  .24%  of 
non-NSPS.  The  2008  NSPS  and  non-NSPS  loss  rates  for  employees  were  similar,  with  NSPS 
at  13.2%  and  non-NSPS  at  12.6%.  The  overall  NSPS  loss  rate  in  2008  was  3.4%. 

Summary: 

DCPDS  analyses  show  that  fewer  than  3%  of  all  ratings  in  2007  and  2008  were  1  or  2,  though 
these  rates  were  many  times  higher  than  for  non-NSPS  systems.  Focus  group  employees  were 
concerned  about  the  impact  of  a  3  rating  upon  their  careers  and  opportunities. 
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5.1.5.  Performance  Expectations  Aligned  with  DoD  and  Organization  Strategic  Plans, 
Mission  Requirements 

SOFS-C 

■  I  know  how  my  work  relates  to  the  Agency's  goals  and  priorities  (Agree/Disagree).  All 

Spiral  1  groups  show  favorable  responses  far  exceeding  unfavorable  responses  on  "I  know 
how  my  work  relates  to  the  Agency's  goals  and  priorities"  in  February  2008. 

■  Managers  communicate  the  goals  and  priorities  of  the  organization  (Agree/Disagree). 

Spiral  1  groups  show  favorable  levels  exceeding  unfavorable  levels,  with  little  movement 
in  attitudes  over  time,  that  managers  communicate  the  goals  and  priorities  of  the 
organization. 

■  NSPS  has  improved  ensuring  individual  performance  supports  organizational  mission 
effectiveness  (Agree/Disagree).  Large  increases  in  unfavorable  responses  since 
November  2006  have  made  favorable  and  unfavorable  response  rates  nearly  equal  for  all 
spirals  in  2008.  The  proportions  of  the  population  who  agree  (about  32%),  disagree 
(about  30%),  and  are  indifferent  (about  38%)  are  similar. 

DCPDS 

■  Over  90  percent  of  sampled  employee  performance  objectives  were  strongly  aligned  and 
very  realistic. 

Focus  Groups  and  Interviews 

■  The  workforce  generally  agreed  that  performance  plans  have  helped  improve  the 
alignment  of  performance  objectives  with  organizational  strategic  goals. 

Summary: 

Findings  from  SOFS-C  and  DCPDS  consistently  show  that  managers  communicate  goals  and 

priorities.  SOFS-C  results  show  increasing  disagreement  concerning  alignment  to  organizational 

goals,  with  focus  group  findings  showing  more  general  agreement  that  alignment  is  occurring. 

5.1.6.  System  Supports  Employee  Engagement  in  a  Positive  Manner  (Employee 
Engagement  Is  Defined  as:  Overall  Satisfaction  with  Job  and  Supervisor,  Respect 
For  Senior  Leaders,  and  Intent  To  Leave  DoD) 

SOFS-C 

■  I  have  trust  and  confidence  in  my  supervisor  (Agree/Disagree).  All  NSPS  spirals' 
favorable  responses  far  exceed  unfavorable  responses  on  having  "trust  and  confidence  in 
my  supervisor,"  as  of  February  2008. 

■  How  satisfied  are  you  with  the  recognition  you  receive  for  doing  a  good  job?  In 

February  2008,  all  NSPS  spirals'  favorable  response  levels  exceeded  unfavorable  levels  on 
being  satisfied  with  the  recognition  they  receive  for  doing  a  good  job. 

■  I  am  given  a  real  opportunity  to  improve  my  skills  in  my  organization  (Agree/Disagree) 
and  How  satisfied  are  you  with  your  opportunities  to  get  a  better  job  in  your 
organization?  All  spirals  more  strongly  agree  than  disagree  in  response  to  "being  given  a 
real  opportunity  to  improve  my  skills."  There  are,  however,  nearly  equal  favorable  and 
unfavorable  response  rates  concerning  satisfaction  with  the  opportunity  to  get  a  better 
job  in  the  organization. 

■  I  have  a  high  level  of  respect  for  my  organization's  senior  leaders  (Agree/Disagree)  and 
Overall,  how  good  a  job  do  you  feel  is  being  done  by  your  immediate  supervisor/team 
leader?  All  NSPS  spirals  have  a  far  higher  agreement  than  disagreement  about  having  a 
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high  level  of  respect  for  organization's  senior  leaders  as  well  as  for  the  performance  of 
their  immediate  supervisor/team  leader. 

■  How  satisfied  are  you  with  the  policies  and  practices  of  your  senior  leaders?  All  NSPS 
spirals  have  a  far  higher  agreement  than  disagreement  about  satisfaction  with 
leadership's  policies  and  practices. 

Focus  Groups  and  Interviews 

■  Some  employees  appreciated  wide  pay  band  flexibilities  for  the  opportunities  they  offer. 

Summary: 

SOFS-C  results  show  that  employees  are  substantially  more  favorable  that  unfavorable  to  senior 
leaders'  policies  and  practices,  and  that  they  have  trust  and  confidence  in  both  their 
supervisors  and  in  senior  leaders.  Employee  focus  group  findings  indicate  that  NSPS's  wide  pay 
bands  are  appreciated  for  the  opportunities  they  provide  to  the  employee. 

5.2.  KPP  2:  Agile  and  Responsive  Workforce  and  Management 

5.2.1.  Renewable,  Expandable,  Retractable  (Size)  -  Mission  Critical/Core  Support 
Occupations 

SOFS-C 

■  My  work  unit  is  able  to  recruit  people  with  the  right  skills  (Agree/Disagree).  Favorable 
responses  continue  to  exceed  unfavorable  responses  that  one's  work  unit  is  able  to 
recruit  people  with  the  right  skills. 

■  NSPS  has  improved  hiring  new  employees  (Agree/Disagree).  Unfavorable  response  rates 
have  increased  sharply  in  the  last  two  years  and  are  higher  than  favorable  response  rates 
that  NSPS  has  improved  hiring  new  employees  for  all  three  Spiral  1  populations  in 
February  2008. 

■  How  would  you  rate  the  performance  of  employees  hired  at  your  organization  over  the 
past  two  years?  and  (Supervisors)  How  would  you  rate  the  quality  of  applicants  for  this 
position  (for  which  you  have  personally  hired  in  the  past  two  years)?  Over  half  of 
respondents  said  that  the  performance  of  employees  hired  at  their  organization  in  the 
past  two  years  has  been  about  average,  but  many  more  said  the  performance  was  above 
average  than  said  it  was  below  average.  Half  of  supervisors  said  that  the  quality  of 
applicants  has  been  better  than  usual. 

Focus  Groups  and  Interviews 

■  While  the  hiring  process  is  difficult  and  time-consuming,  it  is  not  reported  to  be  more 
difficult  than  under  previous  systems.  However,  NSPS's  broad  pay  bands  have  made  it 
more  difficult  to  hire  for  specific  pay  ranges  and  specific  specialty  positions  (e.g.,  IT 
professionals). 

■  Senior  leaders  and  supervisors  generally  believe  that  neither  the  quality  of  applicants  nor 
employee  performance  has  changed;  it  may  be  too  early  to  see  any  effects. 

Summary: 

SOFS-C  findings  generally  show  some  increase  in  satisfaction  with  the  performance  and  quality 
of  employees  that  have  been  hired  in  the  past  two  years,  although  focus  groups  of  supervisors 
counter  this  perspective.  All  results  point  to  a  reluctance  to  credit  NSPS  for  any  improvement  in 
hiring  or  the  performance  of  new  hires  to  this  point.  Focus  group  findings  cite  hiring  for  specific 
pay  ranges  and  specialty  positions  as  particularly  difficult  under  NSPS. 


May  15,  2009 


5-5 


5.  Integrated  Findings 


NS  PS  Spiral  1  Evaluation  Report 


5.2.2.  Assignable,  Deployable  (Shape) 

SOFS-C 

■  NSPS  is  better/worse  than  the  previous  system  for  Hiring,  Placement,  and  Promotion. 

All  spirals  are  significantly  higher  in  their  unfavorable  views  that  NSPS  is  worse/much 
worse  for  hiring,  placement,  and  promotion,  with  most  spirals  showing  sharp  declines  in 
believing  it  is  better  in  this  area,  for  May  2007  and  February  2008.  There  was  a  slight 
increase  in  favorable  levels  that  NSPS  is  better  in  this  area  among  two  spirals  in  February 
2008. 

■  NSPS  is  better/worse  than  the  previous  system  for  workforce  shaping.  Spiral  1.1's 
favorable  and  unfavorable  response  rates  were  generally  unchanged  between  May  2007 
and  February  2008,  while  Spiral  1.2's  and  1.3's  unfavorable  response  rates  increased 
significantly.  All  three  spirals  had  higher  unfavorable  views  than  favorable. 

Focus  Groups  and  Interviews 

■  The  five  percent  pay  increase  limit  for  reassignments  discourages  eligible  employees  from 
assuming  supervisory  positions  with  their  higher  workloads.  Employees  predict  that 
current  supervisors  will  seek  non-supervisory  positions  for  the  same  pay  without  the 
additional  responsibility,  or  will  seek  non-supervisory  positions  in  the  next  higher  pay 
band.  Supervisors  cited  similar  concerns  and  noted  examples  of  supervisors  who  left,  or 
are  considering  leaving  their  organizations  because  of  the  workload. 

■  Some  employees  appreciated  wide  pay  band  flexibilities  for  the  opportunities  they  offer, 
while  others  found  broad  pay  bands  complicated  hiring  and  assignment  processes. 

■  Some  supervisors  asserted  that  NSPS  is  not  meeting  hiring  flexibility  expectations  because 
of  a  limited  number  of  pay  bands  or  the  superimposition  of  GS-based  "pay  lanes"  upon 
NSPS  pay  bands.  Other  supervisors  expressed  appreciation  for  the  increased  flexibility  in 
making  assignments. 

Summary: 

SOFS-C  results  show  that  employees  believe  NSPS  is  not  better  than  the  previous  system  for 
hiring,  placement,  and  promotion.  These  SOFS-C  results  are  partially  supported  by  supervisor 
focus  group  findings  which  cite  "pay  lanes"  practices  and  the  five  percent  reassignment  pay 
increase  limit  as  limiting  hiring  flexibilities.  However,  unfavorable  SOFS-C  results  are  partially 
refuted  as  some  supervisors  appreciate  the  increased  flexibility  for  reassignment. 

5.3.  KPP  3:  Credible  and  Trusted  System 

5.3.1.  All  Elements  of  NSPS  Design  Are  Accessible  and  Understandable  (Transparency) 

SOFS-C 

■  The  pay  pool  panel  helps  ensure  that  the  performance  rating  and  payout  process  is 
equitable  in  my  organization  (Agree/Disagree).  More  employees  in  Spiral  1  disagree  than 
agree  that  the  pay  pool  panel  process  helps  ensure  equitable  ratings  and  performance 
payouts. 

■  How  useful  was  the  information  provided  about  rating  outcomes  in  your  pay  pool?  A 

slight  majority  of  all  Spiral  1  employees  found  information  about  rating  outcomes  neither 
useful  nor  useless  in  February  2008. 

■  In  my  most  recent  performance  appraisal,  I  understood  what  I  had  to  do  to  be  rated  at 
different  performance  levels  (Agree/Disagree).  All  Spirals'  unfavorable  response  levels 
increased  strongly  between  after  their  first  NSPS  performance  cycle  ended,  but  favorable 
response  levels  remained  well  above  unfavorable  levels.  Spiral  1.1's  unfavorable 


5-6 


May  15,  2009 


NSPS  Spiral  1  Evaluation  Report 


5.  Integrated  Findings 


response  levels  decreased  after  their  second  performance  cycle  ended  (reflected  in  the 
February  2008  survey). 

■  [Employees  already  under  NSPS]:  Which  of  the  following  is  the  most  important 
additional  NSPS  training  you  would  like  to  see  provided?  Employees  already  under  NSPS 
are  still  concerned  with  learning  more  about  the  pay  pool  panel  process. 

Focus  Groups  and  Interviews 

■  Many  organizations  had  good  communication  with  the  workforce,  all  published  pay  pool 
panel  business  rules,  and  a  majority  published  rating  and  payout  results.  However, 
employees  were  unevenly  aware  of  pay  pool  panel  processes  and  were  not  satisfied  by 
the  lack  of  feedback  or  explanation  of  rating  and  payout  results. 

■  An  issue  related  to  pay  pool  panel  transparency  is  perceived  differences  in  processes  and 
results  across  pay  pools,  raising  concerns  about  unfair  advantage  and  reward.  PRAs  are 
concerned  with  ensuring  that  pay  pool  panels  are  fair,  transparent,  and  equitable,  and 
were  generally  satisfied  with  the  processes  and  outcomes. 

Summary: 

Findings  from  employee  focus  groups  identify  pay  pool  transparency  as  an  issue,  supported  by 
SOFS-C  results  showing  that  employees  would  like  more  training  on  pay  pool  panel  processes. 
Though  focus  group  findings  show  that  all  organizations  published  business  rules  and  PRAs 
were  generally  satisfied  with  pay  pool  panel  processes  and  outcomes,  SOFS-C  results  show  that 
employees  do  not  see  that  the  pay  pool  panel  process  ensures  equitable  ratings  and 
performance  payouts,  citing  differences  in  processes  and  results  across  pay  pools.  SOFS-C  and 
focus  group  findings  disagree  concerning  whether  employees  understand  what  is  required  to 
be  rated  at  a  different  performance  level. 

5.3.2.  Management  Decisions  Are  Merit-Based,  Non-Discriminatory,  and  Consistent 
With  NSPS  Regulations 

SOFS-C 

■  Promotions  in  my  work  unit  are  based  on  merit  (Agree/Disagree).  All  NSPS  spirals  had 
more  favorable  and  less  unfavorable  responses  that  promotions  in  their  work  units  are 
based  on  merit. 

■  [Employees  already  under  NSPS]:  Which  of  the  following  is  the  most  important 
supervisor  skill  under  NSPS?  Making  fair  personnel  decisions"  was  the  second  greatest 
concern  to  employees  already  under  NSPS,  after  communicating  performance 
expectations. 

■  NSPS  is  better/worse  than  the  previous  system  for  Performance  Management.  There 
were  slightly  higher  unfavorable  response  rates  by  all  three  Spiral  1  populations  that  NSPS 
is  better  for  performance  management  than  their  previous  personnel  system,  with  large 
increases  in  unfavorable  responses  and  slight  increases  in  favorable  responses  between 
May  2007  and  February  2008. 

Summary: 

SOFS-C  results  show  employees  generally  believe  that  promotions  in  their  organization  are 
merit-based.  However,  SOFS-C  results  on  NSPS  performance  management  show  employees' 
views  were  more  unfavorable  than  favorable,  with  recent  sharp  increases  in  unfavorable  views, 
that  NSPS  is  better  than  their  previous  system  for  performance  management.  Also,  SOFS-C 
results  showed  that  NSPS  employees  are  concerned  about  supervisors  making  fair  personnel 
decisions. 
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5.3.3.  Expeditious  and  Fair  Resolution  of  Grievances,  Performance  Reconsiderations, 
Classification  Appeals,  Adverse  Action  Appeals 

DCPDS 

■  Of  the  100,465  NSPS  employees  who  received  a  performance  rating  for  FY2007,  2.3% 
requested  a  reconsideration  of  rating  in  January  2008.  Overall,  a  third  of  the 
reconsideration  requests  were  granted. 

■  In  FY2007  22  NSPS  EEO  complaints  were  received,  and  154  in  FY2008.  A  comparison  of 
2008  complaints  showed  that  the  distributions  by  basis  match  closely  for  NSPS  and  non- 
NSPS,  despite  the  smaller  NSPS  number  of  complaints.  From  this  perspective,  there  is  no 
indication  that  NSPS  is  affecting  how  complaints  are  distributed  across  the  eight  basis 
categories 

Focus  Groups  and  Interviews 

■  There  was  mixed  knowledge  about  and  utilization  of  the  reconsideration  process.  Many 
took  a  "wait  and  see"  attitude  about  NSPS  before  pursuing  reconsiderations.  Some 
employees  believed  that  the  time  limit  for  reconsideration  elapsed  before  employees 
received  pay  pool-  or  organization-wide  feedback  on  ratings  and  payouts.  Other 
employees  simply  did  not  understand  how  to  request  a  reconsideration.  Senior  leaders 
interpreted  the  low  reconsideration  requests  to  indicate  employee  satisfaction  with 
ratings  and  payouts,  but  also  acknowledged  a  lack  of  guidance  regarding  the 
reconsideration  process  as  a  drawback  to  a  well-conducted  process. 

Summary: 

Rating  and  payout  analysis  results  show  a  2.3%  reconsideration  request  rate  for  the  January 
2008  payout  results,  and  interview  findings  indicate  senior  leaders  consider  this  low  rate  to 
suggest  workforce  satisfaction  with  ratings.  However,  employee  focus  group  findings  show  that 
many  in  the  workforce  decided  not  to  request  a  reconsideration,  preferring  to  wait  for  greater 
clarity  on  NSPS  processes,  practices,  and  outcomes. 

5.3.4.  Handle  Performance  Deficiencies  and  Misconduct  Timely  and  Decisively 

SOFS-C 

■  In  my  work  unit,  steps  are  taken  to  deal  with  a  poor  performer  who  cannot  or  will  not 
improve  (Agree/Disagree).  All  NSPS  spirals  had  more  favorable  than  unfavorable 
responses  that  steps  are  taken  in  their  work  units  to  deal  with  a  poor  performer  who 
cannot  or  will  not  improve,  with  unfavorable  responses  declining  between  May  2008 
and  February  2008. 

■  NSPS  has  improved  disciplining/correcting  poor  work  performance  (Agree/Disagree). 

With  the  exception  of  Spiral  1.1,  unfavorable  response  rates  increased  substantially,  and 
all  spirals  remain  at  higher  levels  of  unfavorable  than  favorable  responses  that  NSPS  has 
improved  disciplining/correcting  poor  work  performance. 

■  NSPS  is  better/worse  than  the  previous  system  for  conduct  and  discipline.  All  spirals 
were  mostly  ambivalent  that  NSPS  is  better  or  worse  than  the  previous  system  for 
conduct  and  discipline,  with  relatively  small  levels  of  both  favorable  and  unfavorable 
responses.  Unfavorable  response  rates,  however,  increased  strongly  between  May  2007 
and  February  2008. 

Focus  Groups  and  Interviews 

■  Senior  leaders  (e.g.,  Commanders,  Directors,  and  PRAs)  reported  that  NSPS  is  resulting  in 
a  more  focused  workforce  and  allows  increased  ability  to  address  poor  performance. 
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They  were  also  cautiously  optimistic  that  NSPS  can  eliminate  poor  performers,  if  properly 
instituted. 

Summary: 

Supervisors  and  managers  have  difficulty  in  dealing  with  poor  performers.  Many  employees 
observed  that  marginal  performers  are  seldom  given  2  ratings,  and  senior  leaders  and 
supervisors  generally  agree  that  there  is  institutional  difficulty  in  giving  2  ratings..  While 
responses  on  the  SOFS-C  were  generally  positive  about  steps  taken  in  their  work  unit  to  deal 
with  a  poor  performer,  employees  were  unwilling  to  credit  NSPS  with  improving  such 
situations.  Unfavorable  SOFS-C  responses  concerning  NSPS'  improving  disciplining/correcting 
poor  work  performance  exceed  favorable  ones.  All  spirals  expressed  general  ambivalence 
concerning  whether  NSPS  is  better  or  worse  than  their  previous  system  for  dealing  with 
conduct  and  discipline. 

5.3.5.  NSPS  Employees  in  Aggregate  Are  Not  Disadvantaged  by  the  Compensation 
System 

DCPDS 

■  Rating  and  payout  data  indicate  higher  payouts  for  higher-paid  employees.  Data 
supporting  this  statement  show  that  the  higher  the  base  pay:  1)  the  higher  the  rating,  2) 
the  higher  the  proportion  getting  the  higher  number  of  shares  for  ratings  of  3  or  4,  3)  the 
higher  the  percent  who  received  an  increased  rating  due  to  contributing  factors,  and  4) 
the  higher  the  payout  percentage. 

Focus  Groups  and  Interviews 

■  The  workforce  generally  appreciated  the  value  of  their  payouts  despite  some 
dissatisfaction  with  their  performance  ratings.  However,  employees  at  all  levels 
expressed  concerns  about  the  inclusion  of  part  of  GPI  in  pay  pool  performance  budgets, 
the  effects  of  bonuses  on  retirement  "high  three"  calculations,  and  how  they  fared 
relative  to  GS  employees. 

Summary: 

Rating  and  payout  analyses  suggest  that  payouts  are  relatively  higher  for  higher-paid 
employees.  The  focus  groups  indicated  that  employees  are  concerned  about  including  part  of 
the  GPI  in  pay  pool  performance  budgets,  the  effects  of  bonuses  on  retirement  "high  three" 
calculations,  and  how  they  fared  relative  to  GS  employees. 

5.4.  KPP  4:  Fiscally  Sound  System 

System  Provides  for  Cost  Discipline 

DCPDS 

■  In  aggregate,  the  pay  pools  paid  out  3.1%  less  in  salary  increases  than  the  combined 
funded  amounts  from  Element  1  and  Element  2.  The  difference  largely  reflects  non-rated 
individuals  who  were  not  eligible  for  payouts.  Bonus  payout  exceeded  Element  3  by  5.5%. 
The  total  payouts  were  less  than  two-tenths  of  one  percent  lower  than  the  funding 
amount. 

Focus  Groups  and  Interviews 

■  Some  employees  in  organizations  that  enriched  funding  to  their  NSPS  pay  pools 
questioned  sustainability  over  the  long  term. 
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Summary: 

Some  employees  in  organizations  that  enriched  funding  to  their  NSPS  pay  pools  question 
whether  the  higher  funding  levels  can  be  sustained  over  the  long  term.  In  aggregate,  pay  pools 
under  spent  the  funding  for  salary  increases  (Element  1  and  Element  2)  by  3.1  percent  and 
overspent  the  funding  for  bonuses  (Element  3)  by  5.5  percent.  The  under  spending  of  Element  1 
largely  reflects  non-rated  individuals  who  were  not  eligible  for  payouts.  The  total  payouts  were 
lower  by  two-tenths  of  one  percent  of  the  funding  amount. 

5.5.  KPP  5:  Supporting  Infrastructure 

Employees  and  Managers  Are  Knowledgeable  in  Pertinent  Aspects  of  NSPS  as  a  Result  of 
Training 

SOFS-C 

■  How  useful  was  the  NSPS  training  you  received  on...  (Useful/Very  useful).  Generally,  all 
spirals  have  positive  and  stable  responses  regarding  the  usefulness  of  NSPS  training 
courses.  Employees  would  most  like  to  see  more  training  on  the  performance  appraisal 
system. 

■  [Employees  already  under  NSPS]:  Which  of  the  following  is  the  most  important 
additional  NSPS  training  you  would  like  to  see  provided?  NSPS  employees  wish  to  learn 
more  about  the  performance  appraisal  system  and  pay  pool  panel  process. 

■  [Supervisors  of  NSPS  employees]:  I  effectively  answered  subordinates'  questions  about 
NSPS  performance  rating,  shares,  and  payout  distribution  (Agree/Disagree).  As  of 

February  2008,  supervisors  in  Spiral  1  were  strongly  confident  in  their  abilities  to  answer 
employees'  questions  concerning  NSPS. 

■  [Supervisors  of  NSPS  employees]:  The  rating  assessment  input  I  provided  for  my 
employee(s)  was  effective  in  supporting  the  rating  I  recommended  (Agree/Disagree). 

Supervisors  were  very  confident  in  their  abilities  to  fully  support  their  ratings 
recommendations. 

■  [Supervisors  of  NSPS  employees]:  I  have  the  tools,  training,  and  information  I  need  to 
make  pay  decisions  under  NSPS  (Agree/Disagree).  There  was  a  sharp  increase  in 
agreement  by  supervisors  in  all  NSPS  spirals  from  May  2007  to  February  2008  that  they 
have  the  tools,  training,  and  information  they  need  to  make  pay  decisions  under  NSPS.  As 
well,  favorable  response  levels  greatly  exceeded  unfavorable  response  levels  for  all  Spiral 
1  supervisors  under  NSPS  during  that  time  period. 

Focus  Groups  and  Interviews 

■  Leadership  and  the  workforce  received  varying  levels  of  NSPS  training,  with  differing 
levels  of  effectiveness.  The  lag  time  between  training  and  actual  use  of  skills  was 
problematic,  and  performance  objective  training  was  cited  as  generally  ineffective.  Mock 
pay  pools  were  consistently  praised  as  helpful.  Some  "soft  skill"  instruction  and 
contractor-supported  training  proved  unevenly  effective.  The  continuous,  piecemeal 
distribution  of  NSPS-related  e-mails  containing  updated  information  led  to  information 
overload  and  frustrated  a  comprehensive  understanding  of  NSPS. 

■  Most  employees  and  some  supervisors  desire  a  better  understanding  of  the  pay  pool 
panel  process. 

■  Employees  at  all  levels  desire  more  training  on  writing  objectives  and  assessments. 
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Summary: 

Findings  from  SOFS-C  results  showing  employees  would  most  like  to  see  more  training  on  the 
performance  appraisal  and  pay  pool  panel  processes  concur  with  employee  and  supervisor 
focus  group  findings.  Employees  found  some  "soft  skill"  and  contractor-supported  training 
unhelpful,  though  poor  timing  and  inconsistent  approaches  may  partly  explain  the  lack  of 
effectiveness.  According  to  SOFS-C  results,  supervisors  show  a  strong  increase  in  their 
confidence  to  answer  employees'  questions  concerning  NSPS,  to  support  their  ratings 
recommendations  fully,  and  to  have  the  tools,  training,  and  information  they  need  to  make  pay 
decisions  under  NSPS.  Managers  and  supervisors  found  mock  pay  pool  training  helpful. 

5.6.  Additional  Topics 

5.6.1.  Amount  of  Time  Required  To  Perform  Process 

Focus  Groups  and  Interviews 

■  The  time  and  administration  requirements  of  the  NSPS  performance  management 
system,  particularly  during  the  first  year,  are  a  key  concern  of  leadership  and  the 
workforce.  Many  questioned  whether  NSPS  warrants  the  current  investment  of  time  and 
effort  and  whether  time  requirements  are  sustainable  for  the  long  term.  Senior  officials 
said  it  was  too  soon  to  tell  whether  there  are  offsetting  benefits. 

Summary: 

Focus  group  findings  show  concern  and  uncertainty  whether  the  time  and  administration 
requirements  of  NSPS  will  be  worthwhile. 

5.6.2.  Operation  and  Administration  of  Implementation  Team 

Focus  Groups  and  Interviews 

■  Implementation  teams  were  instrumental  to  implementation  success.  However, 
implementation  efforts  were  sometimes  complicated  by  shifting  policies,  particularly  in 
2006  for  Spiral  1.1  organizations  due  to  NSPS  startup  and  the  short  schedule  and  in  2007 
due  to  the  NDAA  decision  on  Elements  1  and  2. 

Summary: 

Supervisors  and  employees  overwhelmingly  expressed  that  all  aspects  of  NSPS  were  time¬ 
intensive  in  the  first  year.  Focus  group  findings  show  concern  and  uncertainty  whether  the  time 
and  administration  requirements  of  NSPS  will  be  worthwhile. 

5.6.3.  Effects  of  NSPS 

SOFS-C 

■  The  people  I  work  with  cooperate  to  get  the  job  done  (Agree/Disagree).  All  NSPS  spirals' 
favorable  response  levels  far  exceed  unfavorable  levels  indicating  that  people  cooperate 
to  get  the  job  done." 

Focus  Groups  and  Interviews 

■  It  is  too  early  for  most  senior  leaders  to  evaluate  the  effects  of  NSPS  on  individual  and 
organizational  performance.  Among  employees  at  all  levels,  there  is  awareness  that  NSPS 
is  still  taking  root  and  that  it  will  require  time,  effort,  training,  and  leadership  to  fully 
reach  its  promised  potential  and  for  leadership  to  become  fluent  in  its  application.  Other 
employees  have  a  contrary  perspective. 

■  The  workplace  was  alleged  to  have  become  more  competitive  and  less  collaborative 
because  of  NSPS's  focus  on  individual  achievement. 
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Summary: 

While  focus  groups  are  divided  on  whether  the  workplace  is  more  collaborative  or  competitive 
under  NSPS,  SOFS-C  results  show  that  more  employees  feel  their  co-workers  cooperate  to  get 
the  job  done.  Employee  and  supervisor  focus  groups  as  well  as  senior  leader  interview  findings 
suggest  that  all  groups  believe  NSPS  implementation  and  acceptance  will  take  time,  effort, 
training,  and  leadership. 

These  findings  are  often  inconsistent  across  data  sources.  This  result  is  not  entirely  surprising 
as  the  data  sources  measure  different  concepts  in  different  ways.  The  DCPDS  data  reflect 
empirical  results  for  all  of  NSPS.  The  SOFS-C  results  are  statistical  inferences  of  perceptions 
from  a  sample  of  NSPS  employees  (as  well  as  non-NSPS  civilians).  The  focus  groups  are 
protocol-driven  discussions  with  NSPS  employees  and  supervisors  about  their  experiences  and 
perceptions,  and  the  interviews  are  with  senior  managers  and  executives.  Given  the  stage  of 
NSPS  during  which  these  data  collection  efforts  occurred,  it  is  certainly  too  early  to  say  that 
NSPS  has  achieved  the  KPPs,  but  it  is  also  unreasonable  to  say  definitively  whether  NSPS  is  on 
or  off  track.  There  are  signs  of  progress  and  other  signs  of  needed  change. 

5.7.  Recommendations 

Many  of  the  study's  recommendations  address  communication.  Without  effective 
communication  and  training,  NSPS  cannot  achieve  its  goal  of  being  a  credible  and  trusted 
system.  Implementation  teams  were  conscientious  in  their  communication  efforts  -  so  much 
so  that  some  employees  reported  being  overloaded  with  frequent,  lengthy  e-mails  that  they 
didn't  have  time  to  read.  Further,  NSPS  as  a  new  system  continues  to  change,  and  some 
policies,  such  as  those  governing  use  of  contributing  factors,  vary  in  their  implementation. 
Policy  changes,  tool  updates,  new  employees  and  supervisors,  rotating  military  supervisors,  and 
lingering  questions  among  those  who  have  experienced  NSPS  all  call  for  high  levels  of  ongoing 
training  and  communication.  Key  recommendations  include: 

■  More  training  on  the  performance  management  system,  especially  writing  SMART 
objectives,  working  with  supervisors  to  tailor  objectives  that  apply  broadly  to  groups  or 
tasks,  changing  objectives  to  keep  them  aligned  with  changing  assignments,  writing  self- 
assessments,  and  understanding  use  of  performance  indicators  and  benchmarks. 
Supervisors  also  require  training  on  developing  objectives  and  writing  assessments  that 
enable  them  to  meet  the  needs  of  the  pay  pool  panel  and  defend  their  ratings.  Training 
must  include  hands-on  objective  writing  that  is  specific  to  the  participant's  job. 
Inventories  of  SMART  objective  examples  that  cover  a  range  of  job  types  and  levels  have 
proved  helpful.  Best  practices,  such  as  keeping  logs  or  journals  to  document 
accomplishments  and  results,  should  be  shared. 

■  High-level  training  for  employees  and  supervisors  that  explains  the  pay  pool  process, 
including  business  rules  and  other  checks  that  ensure  process  compliance  and 
accountability.  This  training  should  extend  through  the  PRA's  review  and  approval  of 
results.  It  should  also  include  a  high-level  description  of  the  Pay  Pool  Analysis  Tool  (PAT) 
that  allows  the  PRA  to  examine  the  organization's  rating  and  payout  results  across  pay 
pools  and  by  career  group,  pay  schedule,  and  pay  band.  The  workforce  should  also  know 
that  the  components,  as  well  as  the  NSPS  PEO,  conduct  analyses  of  ratings  and  payouts 
for  protected  groups. 

■  Easy  reference  guides,  check  lists,  process  flow  graphics,  time  lines,  and  other  learning 
aids  that  employees  can  keep  at  hand  and  refer  to  quickly. 
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■  Mock  pay  pools,  which  participants  have  found  valuable,  as  refreshers  for  continuing 
panel  members  and  as  training  for  new  members.  This  training  and  other  panel  planning 
meetings  led  by  the  pay  pool  manager  are  important  for  fostering  a  common 
understanding  and  application  of  the  performance  indicators  and  benchmarks,  including 
how  expectations  should  rise  with  the  level  of  work.  This  understanding  of  ratings  and 
performance  indicators  must  be  shared  with  the  managers  and  supervisors  whose 
employees  constitute  the  pay  pool,  as  well  as  with  employees  themselves. 

■  Improved  feedback  for  effective  performance  management.  Managers  and  supervisors 
need  coaching  in  how  to  deliver  useful  feedback,  especially  during  the  interim  and  year- 
end  reviews,  but  also  in  response  to  successes,  failures,  or  missed  opportunities 
whenever  they  occur.  Further,  employees  should  be  encouraged  to  seek  feedback  if  they 
need  but  don't  receive  it. 

■  Greater  clarity  on  career  progression  and  compensation  policy  to  eliminate  confusion  for 
the  workforce  that  has  moved  from  the  comfortable,  well-understood  GS  structure  to 
NSPS  pay  bands.  Policies  or  business  rules  governing  reassignments  that  merit  up  to  a 
five  percent  increase,  where  control  points  apply,  how  performance  expectations  change 
as  the  level  of  work  increases,  rounding  rules,  and  salary-bonus  splits  all  merit  wide 
dissemination  and  discussion.  These  are  issues  on  which  questions  of  fairness  invariably 
arise  if  employees  sense  that  there  is  no  policy  or  that  the  policy  is  complied  with 
selectively.  Among  these,  the  five  percent  limit  on  pay  increases  for  reassignments  is 
perhaps  most  opposed  or  misunderstood. 

5.8.  Conclusion 

The  KPPs  set  high  standards  for  NSPS,  but  certainly  standards  that  an  effective,  performance- 
based  personnel  system  should  achieve.  NSPS  implementation  made  great  progress  from  its 
beginning  in  April  2006  with  just  over  10,000  employees  to  the  conclusion  of  Spiral  1  with  a 
total  of  more  than  100,000.  Program  managers,  implementation  teams,  and  trainers  across 
DoD  worked  with  executives,  managers,  supervisors,  and  employees  to  make  NSPS  real.  DoD 
knows  from  its  decades  of  experience  with  performance-based  demonstration  projects  that 
implementing  and  adjusting  to  change  take  time,  especially  when  the  change  is  of  such 
magnitude  as  moving  many  thousand  employees  from  the  old  but  familiar  General  Schedule  to 
a  new  pay-for-performance  personnel  system.  Evaluations  of  DoD  demonstration  projects  have 
shown  that  negative  initial  responses  become  positive  overtime  as  new  employees  who  accept 
pay  for  performance  join  the  workforce,  those  who  most  resist  it  leave,  processes  become 
more  familiar  and  efficient,  and  high  performers  see  real  rewards.  This  Spiral  1  evaluation 
report  has  examined  NSPS  after  just  two  performance  cycles  for  Spiral  1.1  and  only  one  cycle 
for  Spirals  1.2  and  1.3.  The  report  describes  a  beginning  and  a  process,  not  an  arrival. 

Even  with  massive  training  and  communication  efforts,  the  workforce  has  unanswered 
questions.  Employees  and  supervisors  both  want  to  understand  the  performance  management 
system  better,  especially  what  happens  in  pay  pools  at  the  cycle's  critical  end.  The  vast 
majority  are  invested  in  their  work  and  want  feedback  to  help  improve  their  performance.  The 
workforce  found  the  process  of  writing  SMART  objectives  and  assessments  new  and 
challenging,  and  pay  pool  panels  wrestled  with  performance  plans  that  were  not  always  as 
specific  as  required.  Together,  they  made  it  work,  though  generally  not  as  smoothly  as  anyone 
would  have  wished.  There  were  changes  along  the  way.  Some  changes  were  the  result  of 
learning,  some  the  result  of  Congressional  decision  making  -  all  were  in  the  spirit  of 
improvement. 
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The  way  is  paved  for  the  future.  Leadership  is  on  board,  the  training  is  established,  the  trainers 
are  trained,  and  the  Readiness  Tool  documents  lessons  learned.  Sustainment  training  will 
update  and  refresh  the  original  training,  improve  proficiency,  and  answer  such  lingering 
questions  as  whether  to  weight  objectives,  how  pay  pool  funding  compares  to  GS,  and  how 
business  rules  inform  share  allocation.  The  processes  that  make  up  the  annual  cycle  will 
become  more  familiar.  The  advantages  of  working  with  pay  bands  -  classification, 
reassignments,  career  progression  -  will  become  more  apparent  the  legacy  of  the  GS  structure 
fades.  As  the  newness  continues  to  wear  off,  NSPS  has  the  potential  to  demand  less  time  and 
effort  and  become  a  valuable  tool  for  both  supervisors  and  employees.  It's  on  its  way,  and 
future  evaluations  will  document  its  path. 
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The  2007  NSPS  analysis  also  included  the  following  performance  and  payout  topics: 

Extraordinary  Pay  Increase  (EPI)  and  Organizational  /  Team  Achievement 

Recognition  (OAR) 

NSPS  includes  provisions  for  special  salary  increases  or  lump  sum  bonuses  that  can  be  paid  as  a 
part  of  the  performance  payout  process.  This  is  viewed  as  a  way  to  recognize  exceptional 
performance.  While  this  capability  was  used  by  only  one  pay  pool  in  the  first  cycle  to  give  a 
$3,500  EPI  salary  increase  to  a  single  employee,  Table  A.l  shows  expanded  OAR  and  EPI  usage 
in  the  second  cycle.  There  were  32  EPI  salary  increases  and  6  EPI  bonuses.  The  OAR  was  used 
more  frequently  with  62  OAR  salary  increases  and  352  OAR  bonuses.  The  salary  and  bonus 
maximum  and  average  percentages  are  based  on  base  salary. 


Table  A.l-  2007  EPI  and  OAR  Usage 


EPI  Salary 

OAR  Salary 

Componen 

t 

Number 

Average 

Dollar 

Value 

Maximum 

Dollar 

Value 

Average 

Percent 

Maximum 

Percent 

Number 

Average 

Dollar 

Value 

Maximum 

Dollar 

Value 

Average 

Percent 

Maximum 

Percent 

Army 

3 

$5,395 

$8,877 

7.9% 

13.4% 

52 

$1,786 

$6,000 

1.9% 

10.6% 

Navy 

6 

$6,300 

$8,405 

7.2% 

12.0% 

4 

$950 

$1,550 

1.5% 

2.8% 

Air  Force 

21 

$6,451 

$16,990 

9.6% 

43.5% 

6 

$2,232 

$2,458 

3.7% 

4.0% 

Fourth 

2 

$9,151 

$10,000 

11.1% 

14.2% 

0 

$0 

$0 

0% 

0% 

Estate 

Overall 

32 

$6,492 

$16,990 

9.1% 

43.5% 

62 

$1,776 

$6,000 

2.1% 

10.6% 

EPI  Bonus 

OAR  Bonus 

Componen 

Number 

Average 

Maximum 

Average 

Maximum 

Number 

Average 

Maximum 

Average 

Maximum 

t 

Dollar 

Dollar 

Percent 

Percent 

Dollar 

Dollar 

Percent 

Percent 

Value 

Value 

Value 

Value 

Army 

2 

$8,000 

$8,000 

7.7% 

8.1% 

162 

$1,294 

$3,000 

1.5% 

4.4% 

Navy 

0 

$0 

$0 

0% 

0% 

161 

$507 

$1,634 

0.7% 

2.0% 

Air  Force 

4 

$5,173 

$8,350 

7.4% 

10.0% 

29 

$946 

$2,892 

1.4% 

3.0% 

Fourth 

0 

$0 

$0 

0% 

0% 

0 

$0 

$0 

0% 

0% 

Estate 

Total 

6 

$6,116 

$8,350 

7.5% 

10.0% 

352 

$906 

$3,000 

1.1% 

4.4% 

Impact  of  Five  Percent  Extensions  to  Selected  Pay  Bands 

In  Table  A.2,  we  can  see  that  NSPS  provided  for  a  five  percent  extension  to  the  pay  range  of 
selected  pay  schedules  and  pay  bands.  In  general,  the  Professional  pay  schedule  within  a  career 
group  (YA,  YD,  YH,  and  YK)  saw  the  upper  limit  expanded  by  five  percent  for  those  in  Pay  Band 
3.  For  the  supervisory  pay  schedule  within  each  career  group,  the  upper  limit  was  expanded  for 
those  in  both  Pay  Bands  2  and  3.  For  the  Pay  Band  2  supervisors,  the  maximum  pay  was 
increased  by  5  percent  from  that  of  a  GS-14,  Step  10.  For  all  of  the  Pay  Band  3  supervisors  and 
professionals,  the  maximum  pay  was  increased  by  5  percent  over  that  of  a  GS-15,  Step  10. 
Table  A.2  shows  the  extent  to  which  these  expanded  limits  were  used. 


Table  A.2  -  Utilization  of  Five  Percent  Expansion  in  Certain  Bands 


Group 

Population 

In  5  Percent  Extension 
Before  Payout 

In  5  Percent  Extension 
After  Payout 

Employees 

Percent 

Employees 

Percent 

Band  2  employees  with  5 
percent  bump  out 

17,623 

307 

1.7% 

422 

2.4% 

Band  3  employees  with  5 
percent  bump  out 

16,642 

1,634 

9.8% 

1,881 

11.3% 

Total 

34,265 

1,941 

5.7% 

2,303 

6.7% 
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Use  of  Pro-Ration 

The  pay  pools  have  the  ability  to  apply  pro-ration  to  a  bonus  or  salary  increase.  In  effect,  if  the 
pay  pool  decides  that  a  bonus  or  salary  increase  is  too  high,  it  can  be  reduced  by  applying  a  pro¬ 
ration  value.  For  example,  if  a  bonus  is  computed  to  be  $4,000,  the  pay  pool  can  decide  to 
apply  a  50  percent  pro-ration  factor  and  reduce  the  bonus  to  $2,000.  Due  to  computational 
complexity,  any  reductions  due  to  pro-ration  result  in  a  loss  of  funding  to  the  pay  pool.  Table 
A.3  shows  the  number  of  individuals  who  were  prorated  and  the  dollars  lost  to  salary  and 
bonus  by  level  of  pro-ration. 


Table  A.3  -  Pro-Ration  of  Salary  Increases  and  Bonuses  in  the  2007  NSPS  Cycle 


Prnratinn 

Salary 

Bonus 

Total  Dollar 
Value  Lost 

Average 

Total  Dollar 
Value  Lost 

Average 

Percent  Entered 

Employees 

Dollar  Value 

Lost 

Employees 

Dollar  Value 

Lost 

<10 

3 

$8,058 

$2,686 

3 

$4,096 

$1,365 

10  <20 

1 

$508 

$508 

0 

$0 

$0 

20<30 

14 

$14,029 

$1,002 

25 

$11,962 

$478 

30  <40 

1 

$530 

$530 

4 

$6,067 

$1,517 

40  <50 

2 

$744 

$372 

4 

$3,669 

$917 

50  <60 

254 

$241,902 

$952 

309 

$161,058 

$521 

60  <70 

3 

$1,208 

$403 

10 

$5,758 

$576 

70  <80 

357 

$200,701 

$562 

381 

$106,230 

$279 

80<90 

3 

$820 

$273 

10 

$2,258 

$226 

90  <  100 

3 

$151 

$50 

5 

$563 

$113 

Total 

641 

$468,651 

$731 

751 

$301,661 

$402 

Spiral  1.1  2006  vs.  2007 

Only  9,102  employees  from  Spiral  1.1  received  NSPS  ratings  in  both  2006  and  2007.  Based  on 
this  small  population,  our  ability  to  provide  useful  information  on  trends  is  very  limited. 
Therefore,  we  limited  our  analysis  to  a  simple  comparison  of  average  rating  and  average  shares 
for  the  two  years.  Table  A.4  shows  a  slight  growth  in  average  rating  and  average  shares  for  this 
population.  At  the  pay  pool  level  results  varied,  with  several  pay  pools  showing  significant 
growth  in  average  rating  and  average  shares  and  some  showing  negative  growth. 

Table  A.4  -  Spiral  1.1  Rating  and  Share  Growth 


3.35  3.43  0.08(2.4%)  2.26  2.41  0.15(6.6%) 

The  size  of  the  NSPS  populations  at  the  end  of  the  second  and  third  cycles  will  provide  the 
opportunity  for  a  more  comprehensive  and  meaningful  trends  analysis  in  2008. 


Average  Shares 


Average  Rating 
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SMART  Objectives 

To  what  extent  did  the  objectives  conform  to  the  specific  aspect  of  the  "SMART"  framework? 

»  Not  Available.  Not  enough  information  provided  to  make  a  rating. 

■  Not  Specific.  Did  not  include  specific  and  observable  actions,  behaviors,  or  achievements; 
objectives  were  written  in  very  general  terms  (e.g.,  generic  job  duties  or  work  activities). 

■  Moderately  Specific.  Included  moderately  specific  and  observable  actions,  behaviors,  or 
achievements;  objectives  were  somewhat  vague  or  broad. 

■  Very  Specific.  Included  very  specific  and  observable  actions,  behaviors,  or  achievements 
that  clearly  identified  key  results  and  outcomes. 

To  what  extent  did  the  objectives  conform  to  the  measurable  aspect  of  the  "SMART" 

framework? 

m  Not  Available.  Not  enough  information  provided  to  make  a  rating. 

■  Not  Measurable.  Did  not  include  a  method  or  procedure  for  assessing  or  verifying  the 
quality  of  the  objective. 

■  Moderately  Measurable.  Included  an  approximate,  vague,  or  imprecise  method  or 
procedure  for  assessing  or  verifying  the  quality  of  the  objective. 

■  Highly  Measurable.  Included  an  exact  and  precise  method  or  procedure  for  assessing  or 
verifying  the  quality  of  the  objective. 

To  what  extent  did  the  objectives  conform  to  the  aligned  aspect  of  the  "SMART"  framework? 

m  Not  Available.  Not  enough  information  provided  to  make  a  rating. 

■  Not  Aligned.  Did  not  demonstrate  a  linkage  to  the  relevant  organizational 
mission/strategic  goals  listed  in  the  performance  plan  or  did  not  list  relevant  goals  in  the 
performance  plan. 

■  Moderately  Aligned.  Demonstrated  a  moderately  clear  linkage  to  the  relevant 
organizational  mission/strategic  goals  listed  in  the  performance  plan. 

■  Strongly  Aligned.  Demonstrated  a  very  clear,  direct,  and  strong  linkage  to  the  relevant 
organizational  mission/strategic  goals  listed  in  the  performance  plan. 

To  what  extent  did  the  objectives  conform  to  the  realistic  aspect  of  the  "SMART"  framework? 

m  Not  Available.  Not  enough  information  provided  to  make  a  rating. 

■  Not  Realistic.  Objective  was  not  realistic  and  attainable  given  available  resources,  span  of 
control,  time,  etc. 

■  Somewhat  Realistic.  Objective  was  fairly  realistic  and  attainable,  but  was  a  stretch 
considering  resources,  timeline,  or  span  of  control. 

■  Very  Realistic.  Objective  was  very  realistic  and  attainable  given  available  resources. 

To  what  extent  did  the  objectives  conform  to  the  timed  aspect  of  the  "SMART"  framework? 

m  Not  Available.  Not  enough  information  provided  to  make  a  rating. 

■  No  time  element.  Did  not  include  timeline  for  accomplishing  the  objective. 

■  Included  time  element.  Included  timeline  for  accomplishing  the  objective. 
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Appropriateness  of  Objectives  for  Band  Level  and  Job  Series 

To  what  extent  did  the  objectives  reflect  the  appropriate  difficulty  and  complexity  for  the  band 

level  of  the  employee? 

■  Not  Available.  Not  enough  information  provided  to  make  a  rating. 

■  Too  Low.  Objective  was  written  at  a  level  that  is  too  low,  easy,  or  simple  for  the  band 
level;  objective  would  be  more  appropriate  for  a  lower  band  level. 

■  Appropriate.  Objective  was  written  at  an  appropriate  difficulty  level  for  the  band  level. 

■  Too  High.  Objective  was  written  at  a  level  that  is  too  high,  difficult,  or  complex  for  the 
band  level;  objective  would  be  more  appropriate  for  a  higher  band  level. 

To  what  extent  did  the  objectives  cover  meaningful  outcomes  for  the  job  series? 

■  Not  Available.  Not  enough  information  provided  to  make  a  rating. 

■  Not  Meaningful  Outcome.  Objective  did  not  reflect  a  meaningful  outcome  for  the  job 
series;  objective  was  either  irrelevant  or  insignificant  to  the  tasks  of  the  job. 

■  Moderately  Meaningful  Outcome.  Objective  reflected  a  moderately  meaningful  outcome 
for  the  job  series;  objective  was  relevant  and  reflected  a  core  work  area  for  the  job  series. 

■  Very  Meaningful  Outcome.  Objective  reflected  a  very  meaningful  outcome  for  the  job 
series;  objective  was  relevant,  key,  and  significant  to  the  job  series. 

Adequacy  of  Performance  Descriptions 

To  what  extent  did  the  employee's  self-assessment  provide  adequate  information? 

m  Inadequate  Information.  Assessment  did  not  describe  whether  the  employee 
accomplished  each  objective  (e.g.,  referred  to  general  performance);  did  not  provide 
examples  of  significant  achievements  from  the  rating  cycle;  did  not  provide  examples  of 
how  the  actions  or  conduct  exemplified  the  contributing  factors,  if  applicable;  did  not 
provide  examples  of  how  the  employee's  behavior  made  a  difference  in  the  outcome  of 
the  job  objective;  and  did  not  detail  any  challenges  the  employee  faced  and  how  he/she 
fared. 

■  Adequate  Information.  Assessment  described  whether  the  employee  accomplished  each 
objective;  provided  examples  of  some  significant  achievements  from  the  rating  cycle; 
provided  an  example  of  how  the  actions  or  conduct  exemplified  the  contributing  factors, 
if  applicable;  provided  an  example  of  how  the  employee's  behavior  made  a  difference  in 
the  outcome  of  the  job  objective;  and  described  some  challenges  the  employee  faced  and 
how  he/she  fared. 

■  Very  Extensive  Information.  Assessment  described  whether  the  employee  accomplished 
each  objective;  provided  several  specific  and  concise  examples  of  the  most  significant 
achievements  from  the  entire  rating  cycle;  specifically  described  how  the  organization 
benefited  because  of  what  the  employee  achieved  or  contributed;  provided  several 
examples  of  how  the  actions  or  conduct  exemplified  the  contributing  factors,  if 
applicable;  provided  several  specific  and  concise  examples  of  how  the  employee's 
behavior  made  a  difference  in  the  outcome  of  the  job  objective;  and  described  in  detail 
any  challenges  the  employee  faced  and  how  he/she  fared. 

To  what  extent  did  the  rating  narrative  justify  the  overall  rating  received? 

m  Not  Available.  Not  enough  information  provided  to  make  a  rating. 

■  Inadequate  Justification.  Narrative  did  not  provide  enough  details  to  support  the  rating; 
did  not  provide  any  examples  of  accomplishments  or  deficiencies;  did  not  focus  on 
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objectives  or  results  (e.g.,  focused  only  on  generic  job  duties);  narrative  represented  a 
very  limited  portion  of  the  rating  period. 

■  Adequate  Justification.  Narrative  provided  a  sufficient  amount  of  detail  to  support  the 
rating;  provided  examples  of  accomplishments  or  deficiencies;  focused  on  specific 
objectives  and  results;  narrative  represented  a  sufficient  amount  of  the  rating  period. 

■  Very  Extensive  Justification.  Narrative  provided  an  extremely  significant  amount  of  detail 
to  support  the  rating;  provided  a  significant  number  of  very  detailed  examples  of 
accomplishments  or  deficiencies;  focused  on  objectives  and  results;  narrative 
represented  the  entirety  of  the  rating  period. 

To  what  extent  did  the  rating  narrative  justify  the  contributing  factors  adjustment  (i.e.,  -1,  0, 
+1)? 

■  Inadequate  Justification.  Narrative  did  not  provide  enough  details  to  support  the 
adjustment  (or  lack  thereof);  did  not  provide  any  examples  of  accomplishments  or 
deficiencies. 

■  Adequate  Justification.  Narrative  provided  a  sufficient  amount  of  detail  to  support  the 
adjustment  (or  lack  thereof);  provided  examples  of  accomplishments  or  deficiencies. 

■  Very  Extensive  Justification.  Narrative  provided  an  extremely  significant  amount  of  detail 
to  support  the  adjustment  (or  lack  thereof);  provided  a  significant  number  of  very 
detailed  examples  of  accomplishments  or  deficiencies. 

Similarity  of  Objectives 

To  what  extent  were  the  objectives  similar  within  a  component  for  the  same  job  series,  pay 
schedule,  and  band  level? 

■  Not  Similar.  Less  than  40%  of  the  performance  plans  contained  job  objectives  that  were 
similar  to  one  another  (e.g.,  reflected  similar  outcomes  or  key  results). 

■  Somewhat  Similar.  40-75%  of  the  performance  plans  contained  job  objectives  that  were 
similar  to  one  another  (e.g.,  reflected  similar  outcomes  or  key  results). 

■  Very  Similar.  Over  75%  of  the  performance  plans  contained  job  objectives  that  were 
similar  to  one  another  (e.g.,  reflected  similar  outcomes  or  key  results). 

Difficulty  of  Job  Objectives 

To  what  extent  did  the  objectives  increase  in  difficulty  across  bands  within  the  same  job  series 
and  component? 

m  Did  Not  Increase  in  Difficulty.  Difficulty  varied  across  band  levels  with  easy  and  difficult 
objectives  in  both  the  lower,  middle,  and  higher  bands. 

■  Varied  in  Difficulty  to  a  Moderate  Extent.  At  least  50%  of  the  performance  plans  at  each 
band  level  were  more  difficult  than  the  majority  of  the  performance  plans  at  the  lower 
band  level. 

■  Varied  in  Difficulty  to  a  Great  Extent.  At  least  75%  of  the  performance  plans  at  each  band 
level  were  more  difficult  than  the  majority  of  the  performance  plans  at  the  lower  band 
level. 
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This  study  was  undertaken  at  the  request  of  the  NSPS  PEO  to  evaluate  the  implementation  and 
results  of  NSPS  Spiral  1.  The  primary  purpose  of  the  study  was  to  evaluate  implementation 
progress  and  identify  areas  where  adjustment  could  be  made  to  ensure  that  NSPS  meets  its 
objectives.  The  primary  research  question  addressed  in  this  report  is  whether: 

■  NSPS  is  on  course  to  meet  the  expectations  set  by  its  key  performance  parameters  (KPPs). 

The  KPPs  specify  the  broad  goals  of  NSPS  and  how  their  achievement  will  be  measured.  The  five 
KPPs  and  their  associated  objectives,  shown  in  Table  C.l,  provide  the  foundation  for  this  study. 

Table  Cl  -  Key  Performance  Parameters 

1.  High  Performing  Workforce  and  Management 

Stronger  link  between  total  compensation  and  performance 
Market  Sensitive  Salary  and  Wages 
Performance  System  Provides  Ongoing  Feedback 
Stronger  link  between  retention  /  separation  and  performance 

Performance  expectations  aligned  with  DoD  and  organization  strategic  plans,  mission  requirements 
System  supports  employee  engagement  in  a  positive  manner 

2.  Agile  and  Responsive  Workforce  and  Management 

Renewable,  expandable,  retractable  (size)  -  Mission  Critical/Core  Support  Occupations 
Assignable,  deployable  (shape) 

Diverse  (multifaceted) 

3.  Credible  and  Trusted  System 

All  elements  of  NSPS  design  are  accessible  and  understandable  (transparency) 

Management  decisions  are  merit-based,  non-discriminatory,  and  consistent  with  NSPS  regulations 

Expeditious  and  fair  resolution  of  grievances  (negotiated/admin),  performance  reconsiderations,  classification  appeals, 

adverse  action  appeals 

Handle  performance  deficiencies  and  misconduct  timely  and  decisively 

Bargaining  process  allows  DoD  and  components  to  implement  policies  quickly  and  consistently 
Resolve  labor  disputes  and  negotiated  grievances  timely 

4.  Fiscally  Sound  System 

System  provides  for  cost  discipline  within  the  Department's  top  line 

5.  Supporting  Infrastructure 

Employees  and  managers  are  knowledgeable  in  pertinent  aspects  of  NSPS  as  a  result  of  training 

The  process  by  which  the  qualitative  (i.e.,  focus  group  and  interview)  data  were  collected  and 
analyzed  is  described  in  this  appendix. 

Development  of  the  Data  Collection  Protocols 

Prior  to  conducting  the  site  visits,  the  project  team  developed  protocols  containing  questions 
designed  to  address  the  KPPs.  To  develop  these  protocols,  the  study  team  worked  with  the  PEO 
to  draft,  refine,  and  evaluate  questions. 

Separate  protocols  were  developed  for  each  of  the  following  six  cohorts: 

■  Employee 

■  Supervisor 

■  Manager/Pay  Pool  Panel  Member 

■  Implementation  Team  Member 

■  Commander/Director 

■  Performance  Review  Authority  (PRA). 
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The  questions  contained  in  these  protocols  were  designed  to  elicit  responses  about  specific 
aspects  of  NSPS  that  were  relevant  to  the  cohort.  Each  protocol  contained  three  types  of 
questions: 

■  Level  1:  High  priority  questions  that  facilitators  were  instructed  to  ask  in  every  focus 
group  or  interview.  These  questions  tended  to  be  directly  relevant  to  the  KPPs. 

■  Level  2:  Additional  questions  that  facilitators  could  ask,  but  were  not  likely  to  ask  in  each 
focus  group  or  interview.  These  questions  tended  to  be  less  directly  relevant  to  the  KPPs, 
but  were  expected  to  provide  valuable  information  about  specific  topics. 

■  Level  3:  Additional  questions  that  facilitators  could  ask  if  the  participants  were  having 
difficulty  answering  related  questions.  These  questions  tended  to  be  follow-up  questions 
that  probed  larger  issues.  They  often  were  answered  in  the  course  of  responding  to  Level 
1  and  Level  2  questions. 

The  draft  protocols  were  pilot  tested  as  part  of  the  first  site  visit  to  ensure  that  the  questions 
were  clear  and  that  the  protocols  could  be  completed  in  the  allotted  time.  Based  on  the  results 
of  the  pilot  test,  no  changes  were  made  to  the  protocols.  Copies  of  the  protocols  are  provided 
in  Appendix  D. 

Collection  of  Focus  Group  and  Interview  Data 

Credentials  of  Study  Team 

A  data  collection  team  consisting  of  a  facilitator  and  a  recorder  conducted  each  focus  group  and 
interview.  A  representative  from  the  NSPS  PEO  and  the  relevant  component  were  also  present 
for  the  Commander/Director,  PRA,  Managers/Pay  Pool  Panel,  and  Implementation  Team 
interviews.  The  PEO  and  component  representatives  provided  an  introduction  to  the  employee 
and  supervisor  focus  groups,  but  were  not  present  during  the  discussions. 

Members  of  the  data  collection  teams  were  responsible  for  facilitating  and  recording  the  focus 
groups.  The  role  of  the  facilitator  was  to  guide  the  discussion  following  the  protocol.  The  role  of 
the  recorder  was  to  create  an  annotated  transcript  of  the  discussion.  Members  of  the  project 
team  alternated  between  facilitating  and  recording. 

Participating  Sample 

The  data  collection  teams  conducted  111  focus  groups  and  interviews  with  approximately  635 
employees6.  Table  C.2  provides  an  overview  of  the  number  of  focus  groups  and  interviews  by 
location  and  cohort. 


Table  C.2  -  Number  of  Focus  Groups  and  Interviews  by  Location  and  Cohort 


Cohort 

Location 

Employee 

Supervisor 

Pay  Pool 
Panel 

Imp. 

Team 

Commander 

/Director 

PRA 

Total 

CHRA  North  Central 

2 

1 

1 

2 

1 

0 

7 

CPOC 

Tripler  Army  Medical 

3 

2 

1 

1 

0 

1 

8 

Center 

Army  Engineering 

3 

2 

2 

1 

1 

1 

10 

District,  Jacksonville 

HQUSMC 

3 

2 

2 

1 

1 

1 

10 

PACFLT 

2 

1 

3 

1 

1 

8 

SPAWAR  SYSCEN 

5 

2 

2 

1 

1 

1 

12 

Scott  AFB 

3 

2 

3 

1 

0 

1 

10 

6  This  number  is  estimated  because  not  all  employees  and  supervisors  completed  a  demographic  questionnaire  and 
demographic  data  were  not  collected  from  senior  leadership. 
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Location 


Offutt  AFB 

Wright-Patterson  AFB 
Defense  Security 
Cooperation  Agency 
Missile  Defense  Agency 
DoDIG 
Total 

Focus  groups  typically  consisted  of  six  to  twelve  participants;  however,  some  focus  groups  were 
conducted  with  as  few  as  three  and  as  many  as  twenty  participants  due  to  variations  in 
employee  and  supervisor  availability  during  the  site  visits.  Interviews  were  typically  conducted 
with  one  to  three  participants.  Demographics  of  the  participating  samples  are  provided  in 
Appendix  F. 

Data  Collection  Procedures 

At  the  start  of  each  session,  the  facilitator  described  the  goals  of  the  overall  study  and  provided 
an  opportunity  for  participants  to  ask  questions.  Focus  group  participants  were  advised  that 
information  obtained  during  the  session  was  to  be  treated  as  non-attributional  and 
confidential.  Each  participant  completed  a  demographic  questionnaire.7  Participants  were 
provided  a  telephone  number  for  the  facilitator  in  the  event  they  wanted  to  express  some 
additional  thoughts  after  the  session  ended. 

After  collecting  the  demographic  questionnaire  the  facilitator  followed  the  relevant  focus  group 
protocol.  In  addition,  the  facilitator  asked  probe  questions  as  necessary  to  gather  further 
information.  Facilitators  used  their  own  discretion  to  select  probe  questions  based  on  the  detail 
and  length  of  the  group's  response.  It  is  important  to  note  that  the  purpose  of  the  focus  groups 
was  to  collect  employees'  perceptions  and  beliefs  regarding  the  questions.  It  is  possible  that 
these  perceptions  and  beliefs  do  not  accurately  reflect  actual  NSPS  processes,  but  rather 
employees'  perceptions  of  these  processes.  To  balance  perceptions  by  focus  group  participants, 
the  research  team  examined  contextual  elements  of  each  organization,  component,  and  DoD  to 
identify  "disconnects"  and  inconsistencies  in  the  implementation  process. 

All  focus  group  participants  were  given  an  opportunity  to  respond  to  each  question;  however, 
because  participation  was  voluntary,  facilitators  did  not  "call  on"  individual  participants  to 
speak.  During  the  entire  focus  group,  the  recorder  captured  detailed  transcripts  of  the 
participants'  responses  to  each  question,  without  any  identifying  information.  To  conclude  the 
focus  group,  the  facilitator  debriefed  all  the  participants.  The  debriefing  covered  the  following 
topics: 

■  A  reminder  that  that  all  information  provided  in  the  focus  group  will  be  kept  completely 
anonymous  and  non-attributional  by  name,  organization,  or  installation.  No  data  will  be 
linked  back  to  any  individual. 

■  The  remaining  schedule  for  the  data  collection  and  analysis  process.  Specifically,  the 
participants  were  told  that  focus  groups  were  being  conducted  throughout  the  selected 
12  sites  and  that  at  the  conclusion  of  the  focus  groups  the  data  collection  teams  will  meet 
to  review  the  results  and  produce  a  report  for  the  NSPS  PEO. 


7  Demographic  data  were  collected  only  during  focus  groups.  Interview  participants  did  not  complete  the  demographic 
questionnaire. 
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■  A  reminder  that  participants  could  send  an  email  to  the  facilitator  if  they  had  any  other 
comments  or  suggestions  they  would  like  to  provide.  This  information  would  also  be 
treated  as  non-attributional  and  would  be  considered  part  of  the  focus  group  discussion. 

Analysis  of  Focus  Group  and  Interview  Data 

Cleaning  and  Analyzing  Individual  Focus  Group  and  Interview  Transcripts 

At  the  conclusion  of  a  focus  group  session,  the  recorder  produced  a  raw  transcript.  This 
document  may  have  contained  typographical  errors,  abbreviations,  and  other  shorthand  that 
was  necessary  for  the  rapid  acquisition  of  the  data  in  real  time.  Next,  the  recorders  cleaned  and 
edited  the  raw  transcript  to  correct  any  typographical  errors  and  fill  in  any  detail  that  was 
missing  due  to  the  use  of  abbreviations  and  shorthand.  Further,  recorders  then  placed  these 
responses  as  appropriate  under  the  pre-defined  questions.  Duplication  of  data  was  allowed  if 
the  data  answered  more  than  one  of  the  pre-defined  questions.  This  corrected  and  sorted 
document,  which  was  stored  separately  from  the  raw  data,  is  referred  to  as  the  cleaned_ 
transcript.  It  is  this  cleaned  transcript  that  is  the  initial  unit  of  analysis. 

Next,  each  facilitator  produced  an  analyzed [  transcript.  The  cleaned  transcript  was  content 
analyzed  to  identify  salient  themes  and  quotations  that  supported  the  themes.  Themes  were 
identified  using  the  following  criteria: 

■  The  prevalence  of  the  concept  in  the  transcript 

■  The  importance  of  the  theme  to  the  overall  research  question 

■  Internal  homogeneity  -  does  the  theme  express  a  unified  idea? 

■  External  heterogeneity  -  is  the  theme  unique  from  other  themes? 

This  process  resulted  in  the  identification  of  approximately  4,000  individual  themes. 

Cross-Transcript  Analysis 

To  identify  and  remove  redundancy  from  the  approximately  4,000  individual  themes,  the  study 
team  next  conducted  a  cross-transcript  analysis  for  each  of  the  six  cohorts  (i.e.,  Employee, 
Supervisor,  Manager/Pay  Pool  Panel  Member,  Implementation  Team  Member, 
Commander/Director,  and  PRA).  This  methodology  quickly  and  effectively  aggregates  themes 
from  multiple  focus  groups. 

To  conduct  this  analysis,  members  of  the  study  team  linked  each  of  the  4,000  themes  to  one  of 
57  larger  theme  categories  and  identified  and  eliminated  redundancies  using  the  criteria  for  a 
theme  described  above.  Next,  the  study  team  participated  in  a  facilitated  discussion  during 
which  we  reviewed  the  reduced  set  of  211  non-redundant  themes  for  each  cohort  for  accuracy. 
While  each  facilitator/recorder  team  was  well  versed  in  the  themes  from  their  respective 
sessions,  no  one  team  had  complete  knowledge  of  overarching  themes.  Therefore,  the 
objective  of  this  analysis  was  to  determine  the  salient  themes  for  each  of  the  six  cohorts. 
Through  this  process,  the  study  team  identified  the  themes  that  emerged  most  consistently 
across  all  of  the  focus  groups. 

Cross-Cohort  Analysis 

To  consolidate  the  211  cohort-specific  themes  into  a  smaller,  more  meaningful  set  of 
overarching  themes,  the  study  team  next  conducted  a  cross-cohort  analysis.  The  cross-cohort 
analysis  methodology  was  essentially  the  same  as  the  cross-transcript  analysis  methodology, 
but  was  used  to  identify  and  eliminate  redundancies  across  cohorts  while  also  retaining  any 
subtle  differences  in  perspective  that  are  relevant  to  particular  cohorts.  These  cross-cohort 
themes  represent  the  strongest,  most  consistent  findings  from  across  all  focus  groups  and 
interviews.  Through  this  process,  the  study  team  clustered  the  cross-transcript  themes  into  16 
cross-cohort  themes. 
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Focus  Group  Protocol  -  Employee 

Thank  you  for  joining  us  today.  My  name  is _ from _ and  I  will  facilitate  this 

focus  group.  I  am  joined  by: 

[Ask  the  visiting  group  to  introduce  themselves  and  to  state  their  organization] 

As  you  know.  Spiral  1  of  the  National  Security  Personnel  System  (NSPS)  covers  more  than 
100,000  employees,  and  70,000  more  are  converting  this  year  in  Spiral  2.  We  are  in  the  process 
of  conducting  focus  groups  and  interviews  in  12  organizations  across  the  country  to  help  us 
evaluate  NSPS  as  it  was  implemented  in  Spiral  1. 

Although _ will  be  taking  detailed  notes  on  the  information  collected  in  this  focus 

group,  all  responses  will  remain  anonymous  and  non-attributional.  That  is,  we  will  not  link  your 
name  to  any  of  the  comments  that  you  make.  The  results  of  this  session  will  be  combined  with 
other  sessions  here  and  elsewhere  as  we  evaluate  NSPS. 

[Hand  out  the  demographic  questionnaire.] 

The  purpose  of  this  form  is  to  collect  basic  information  about  the  makeup  of  this  group,  so  we 
can  think  about  your  responses  as  a  representative  sample  of  everyone  who's  under  NSPS.  The 
data  will  be  stored  separately  from  your  responses  and  will  not  be  linked  to  any  participants' 
comments.  You  don't  have  to  give  us  the  information,  but  it  will  be  helpful  if  you  do. 

If  you  have  issues  you  would  like  to  bring  up,  but  are  not  comfortable  doing  so  in  this  group, 
you  can  contact  me  at _ to  set  up  a  telephone  interview. 

Because  the  information  we  are  collecting  in  this  session  is  so  critical  and  the  time  for  this 
session  is  limited,  we  ask  that  you  please: 

■  Speak  clearly  and  to  the  facilitator 

■  Speak  one  at  a  time 

■  Avoid  side  conversations 

This  session  is  expected  to  last  for  up  to  two  hours.  During  this  session,  we  will  ask  about  your 
experiences  in  NSPS  and  how  NSPS  has  affected  you  and  your  organization.  Before  we  begin,  I'd 
like  for  you  to  introduce  yourselves.  Please  give  us  your  name,  which  organization  you're  with, 
and  how  long  you've  been  here  at _ . 

Thanks  to  all  of  you  for  taking  the  time  to  be  with  us  today. 

Are  there  any  questions? 

Let's  begin  with  some  general  questions. 

1.  What,  if  anything,  feels  different  about  being  in  NSPS? 

2.  What  NSPS  training  did  you  receive,  and  how  helpful  was  it?  [Examples  of  training 
include  NSPS  101,  performance  management  system,  writing  objectives] 

■  What  was  most  (and  least)  helpful  about  the  training? 

■  Are  there  any  topics  or  skills  that  you  think  weren't  covered  adequately  and 
should  be  part  of  future  training? 

3.  How  well  were  you  kept  informed  about  what  to  expect  and  about  how  local  processes 
changed  under  NSPS? 
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Next,  let's  talk  about  your  experiences  with  performance  plans,  objectives,  and  the  rating 
process. 

4.  What  was  your  role  in  developing  your  performance  objectives? 

■  Did  you  write  your  own  objectives,  work  with  your  supervisor  to  write  them, 
comment  on  standard  objectives? 

5.  To  what  extent  did  your  performance  plan  last  year  cover  your  major  work  activities 
with  clear,  reasonable  objectives? 

■  Did  the  contributing  factors  make  sense  for  your  job  ?  What  makes  you  say  this  ? 

■  Did  the  objectives  capture  the  right  criteria  and  measures  to  assess  your 
performance? 

■  Did  your  performance  plan  reflect  outcomes  that  were  important  for  your  job? 

6.  Does  your  performance  plan  for  this  year  make  sense  to  you?  Do  you  understand  what 
it  will  take  to  get  a  rating  of  3  and  higher? 

■  Thinking  back  to  last  year,  how  useful  was  your  supervisor's  feedback  during 
your  interim  review  to  help  you  improve  or  change  your  performance? 

•  Was  the  feedback  specific  enough  so  that  you  knew  what  to  concentrate  on? 

•  How  did  you  change  your  performance— if  at  all— after  getting  your 
supervisor's  feedback  at  interim  and  other  reviews? 

•  Did  you  and  your  supervisor  discuss  your  career  development  during 
performance  reviews? 

■  If  you  completed  a  self-assessment,  how  valuable  do  you  think  it  was? 

7.  To  what  degree  do  you  think  your  performance  was  appropriately  rated? 

■  Can  you  think  of  anything  that  would  make  the  rating  process  better? 

8.  How  clear  were  your  rater's  explanation  of  your  rating  and  responses  to  your  questions? 

■  Was  he  or  she  able  to  answer  your  questions  about  your  ratings  and  payout? 

9.  How  much  information  did  you  get  about  rating  and  payout  results  across  your  pay 
pool? 

■  Was  this  information  helpful? 

Now,  let's  talk  about  the  performance  system  in  general  terms. 

10.  How  well  do  you  understand  your  organization's  pay  pool  process?  Also  comment  on 
how  you  got  information  about  the  process, 

■  For  example:  written  notices  about  the  pay  pool,  training,  and/or  someone  you 
could  turn  to  for  additional  information  to  satisfy  your  questions. 

■  Examples  of  notice  information:  which  employees  are  included,  whether  there 
are  subpools,  the  names  of  the  pay  pool  manager  and  panel  members,  general 
pay  pool  policies  and  business  rules. 

11.  What  safeguards  do  you  think  are  most  important  to  make  sure  the  rating  and  payout 
decisions  are  fair  and  equitable? 

12.  How  strong  a  link  do  you  see  between  performance  and  pay  under  NSPS? 

■  What  about  a  link  between  performance  and  the  size  of  bonuses? 

■  Are  the  links  stronger  under  NSPS  than  they  were  under  your  previous  system  ? 

Now,  let's  move  to  some  other  aspects  of  NSPS. 

13.  What's  your  opinion  of  how  NSPS's  broad  pay  bands  will  affect  your  job  opportunities 
and  assignments  in  the  organization?  What  makes  you  think  this? 
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■  Will  they  affect  the  assignments  you  receive  or  seek  out? 

■  Do  you  know  how  to  advance  in  your  career  under  NSPS? 

14.  Has  NSPS  affected  your  job  satisfaction  or  your  motivation? 

■  Do  the  performance  discussions  with  your  supervisor,  and  the  potential  rewards 
under  NSPS  affect  the  way  you  perform  your  job? 

■  Are  you  starting  to  see  meaningful  differences  in  the  way  performance  at 
different  levels  is  rewarded? 

15.  How  have  things  worked  out  under  NSPS  compared  to  what  you  thought  might  happen? 

■  Do  you  plan  to  do  anything  differently  this  year  as  a  result  of  your  experience 
with  NSPS?  If  so,  what? 

16.  Do  you  think  you  will  be  better  off  in  the  future  under  NSPS?  What  makes  you  say  this? 

■  What  is  the  one  thing  about  NSPS  you  would  most  like  to  change? 

17.  Do  you  believe  that  your  organization  will  be  better  off  under  NSPS?  What  makes  you 
say  this? 

■  What  do  you  think  the  long  term  effects  of  NSPS  will  be? 

Remember,  if  you  have  issues  you  would  like  to  bring  up,  but  are  not  comfortable  doing  so  in 
this  group,  or  if  you  think  of  any  other  comments  you  would  like  to  make,  you  can  contact  me 
at _ to  set  up  a  telephone  interview. 

Thank  you  for  talking  to  us  today. 

Focus  Group  Protocol  -  Supervisors 

Thank  you  for  joining  us  today.  My  name  is _ from _ and  I  will  facilitate  this 

focus  group.  I  am  joined  by 

[Ask  the  visiting  group  to  introduce  themselves  and  to  state  their  organization] 

As  you  know,  Spiral  1  of  the  National  Security  Personnel  System  (NSPS)  covers  more  than 
100,000  employees,  and  another  70,000  are  converting  this  year  in  Spiral  2.  We  are  in  the 
process  of  conducting  focus  groups  and  interviews  at  12  sites  across  the  country  to  help 
evaluate  NSPS  as  it  was  implemented  in  Spiral  1. 

Although _ will  be  taking  detailed  notes  on  the  information  collected  in  this  focus 

group,  all  responses  will  remain  anonymous  and  non-attributional.  That  is,  we  will  not  link  your 
name  to  any  of  the  comments  that  you  make.  The  results  of  this  session  will  be  combined  with 
other  data  to  evaluate  NSPS. 

[Hand  out  the  demographic  questionnaire.] 

The  purpose  of  this  form  is  to  collect  basic  demographic  information  so  we  can  evaluate  the 
representativeness  of  the  participant  sample  relative  to  the  entire  population  that  is  under 
NSPS.  The  data  will  be  stored  separately  from  your  responses  and  will  not  be  linked  to  any 
participants'  responses. 

If  you  have  issues  you  would  like  to  bring  up,  but  are  not  comfortable  doing  so  in  this  group, 
you  can  contact  me  at _ to  set  up  a  telephone  interview. 

Because  the  information  we  are  collecting  in  this  session  is  so  critical  and  the  time  for  this 
session  is  limited,  we  ask  that  you  please: 

■  Speak  clearly  and  to  the  facilitator 

■  Speak  one  at  a  time 

■  Avoid  side  conversations 
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This  session  is  expected  to  last  for  up  to  two  hours.  During  this  session,  we  will  be  asking  you 
about  your  experiences  in  NSPS  and  how  NSPS  has  affected  you  and  your  organization.  Before 
we  begin,  I'd  like  for  you  to  introduce  yourselves.  Please  give  us  your  name,  which  organization 
you're  with,  and  how  long  you've  been  here  at _ . 

Thanks  to  all  of  you  for  taking  the  time  to  be  with  us  today. 

Are  there  any  questions? 

Let's  begin  with  some  general  questions. 

1.  What,  if  anything,  feels  different  about  supervising  employees  under  NSPS? 

2.  What  NSPS  training  have  you  taken,  and  how  helpful  was  it? 

■  Are  the  on  line  NSPS  courses  useful  supplements? 

■  Are  there  any  topics  or  skills  that  you  think  weren't  covered  adequately  and  should 
be  part  of  future  training? 

Next,  we'd  like  to  hear  about  your  experiences  with  hiring,  pay,  and  assignment  aspects  of 
NSPS. 

3.  What's  your  opinion  of  how  NSPS's  broad  pay  bands  affect  job  opportunities  and 
assignments  for  your  and  your  employees?  Expand  upon  this  and  give  examples. 

■  Will  they  affect  the  assignments  you  receive  or  seek  out  for  yourself? 

■  How  do  they  affect  work  assignments  you  give  your  employees? 

■  Are  new  career  paths  starting  to  emerge  under  NSPS? 

■  Have  you  noticed  any  examples  of  NSPS'  broader  pay  bands  and  career  groups 
making  it  easier  to  move  people  or  work  within  your  organization? 

4.  Tell  us  how  NSPS  pay  flexibilities  help  or  don't  help  when  you  go  to  reassign,  relocate,  or 
promote  employees? 

5.  In  your  opinion,  has  NSPS  made  a  difference  in  attracting  and  hiring  high  quality 
candidates?  Expand  upon  your  thoughts  with  examples. 

■  How  does  your  local  pay  policy  or  pay  approval  process  affect  your  pay  offers ? 

•  Have  there  been  any  situations  where  NSPS  broadbands  and  career  groups 
created  difficulties  in  staffing?  Can  you  give  some  examples? 

•  How  would  you  compare  quality  of  applicants  and  the  people  you  hire  under 
NSPS  to  those  under  your  previous  system? 

Now  I'd  like  to  move  on  to  performance  management. 

6.  How  did  you  involve  employees  in  developing  their  performance  plans? 

■  Who  prepared  the  first  draft? 

■  If  you  used  standard  objectives,  did  employees  have  a  role? 

•  Were  all  of  your  employees'  performance  objectives  measurable? 

•  What  questions  did  your  employees  ask  you  about  their  performance 
expectations  after  their  performance  plans  were  in  place? 

•  How  well  did  the  measures  you  included  in  their  objectives  reflect  outcomes  that 
are  important  to  their  jobs? 

■  Did  the  contributing  factors  help  you  focus  the  employee's  attention  on  the  most 
important  aspects  of  how  the  objective  should  be  carried  out? 

7.  What  is  your  opinion  of  the  value  and  effectiveness  of  interim  reviews  with  your 
employees? 

■  Did  you  obtain  any  insight  from  the  employee  about  work  progress  and  products? 
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■  Did  you  identify  any  changes  the  employee  needed  to  make  to  perform  most 
effectively? 

■  Did  you  discuss  your  employee's  career  development? 

m  Do  you  plan  to  do  interim  reviews  and  informal  discussions  differently  in  the  future? 
If  so,  what  will  you  do  differently? 

8.  How  did  you  measure  employee  performance? 

■  Did  your  employees  complete  a  self-assessment? 

•  How  valuable  do  you  think  the  self-assessment  was? 

■  How  useful  were  the  benchmarks  and  performance  indicators  in  rating 
performance?  To  what  extent  were  you  able  to  match  performance  with  the 
performance  indicators  at  the  level  you  felt  was  appropriate? 

■  What  questions  or  comments  did  your  employees  ask  about  how  you  saw  their 
performance? 

■  In  your  assessment  input,  were  you  comfortable  that  you  made  the  case  to  support 
your  employees'  recommended  rating? 

•  What  worked  well?  What  difficulties  did  you  encounter  and  how  did  you  resolve 
them  ? 

•  Which  types  of  objectives  proved  the  hardest  to  measure? 

•  Did  you  credit  contributing  factors  when  you  recommended  ratings? 

■  Do  you  think  the  rating  process  is  fair  to  employees? 

•  To  what  extent  do  you  think  the  performance  you  measured  accurately  reflected 
the  important  things  employees  did  and  how  well? 

•  To  what  extent  were  you  able  to  differentiate  among  performance  levels? 

•  In  your  assessment  input,  were  you  comfortable  that  you  made  the  case  to 
support  your  employees'  recommended  rating? 

9.  What  sorts  of  questions  or  comments  did  your  employees  have  about  the  rationale 
behind  their  ratings? 

■  What,  if  anything,  did  you  do  or  say  that  helped  your  employees  understand  their 
ratings? 

Now  let's  talk  about  some  general  aspects  of  the  performance  system. 

10.  Talk  about  the  information  you  got  from  the  pay  pool  panel  or  subpanel,  and  any 
interaction  you  had  with  them  about  outcomes. 

■  What  information  did  you  get  about  the  business  rules,  funding  levels,  deadlines? 

■  How  did  you  handle  any  pay  pool  panel  or  subpanel  questions  about  your 
assessment  of  employee?  Was  the  interaction  valuable? 

11.  Was  the  performance  rating  and  payout  process  in  your  organization  transparent? 
What  would  have  helped  make  it  transparent? 

12.  How  do  you  make  decisions  about  high  impact  assignments  that  can  give  one  employee 
a  leg  up  when  the  panel  reviews  the  accomplishments? 

■  Do  you  have  any  strategies  for  minimizing  the  appearance  of  favoritism  in  making 
these  assignments? 

13.  How  strong  a  link  do  you  see  between  performance  and  pay  under  NSPS?  That  is,  do 
you  think  your  stronger  performers  are  getting  bigger  payouts  and/or  bonuses  than 
average  performers?  And  are  the  payouts  big  enough  to  matter  to  them? 

■  TO  SPAWAR  and  MDA  supervisors:  Are  these  links  any  stronger  under  NSPS  than 
under  your  former  system  ? 
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■  Do  you  think  higher  performers  will  move  ahead  faster  under  NSPS? 

14.  To  what  extent  has  NSPS  affected  your  ability  to  address  poor  performance,  if  any, 
among  your  employees? 

To  conclude  the  session,  we  would  like  to  ask  you  a  few  general  questions  about  NSPS. 

15.  What  effects  are  you  seeing  on  your  workforce? 

■  Have  you  had  any  employees  leave  or  talk  about  leaving  since  NSPS  started? 

•  How  do  you  think  the  performance  assessment  and  payout  process  affected  their 
decision  or  intention? 

•  Do  you  consider  those  who  left  or  talk  about  leaving  to  be  high  performers, 
average  performers,  or  low  performers? 

■  In  your  opinion,  what  effects  has  NSPS  had  on  how  your  employees  do  their  jobs? 

•  If  you  have  noticed  changes  in  employee  satisfaction  or  motivation,  what  do  you 
think  is  causing  them?  The  potential  for  rewards?  The  increased  flexibility  of  job 
assignments?  Something  else? 

■  Was  the  amount  of  time  spent  on  performance  plans,  discussions,  and  evaluation 
valuable? 

•  Do  you  think  it  will  take  a  similar  amount  of  time  in  the  future? 

■  Do  you  plan  to  do  anything  differently  this  year  as  a  result  of  your  experience  with 
NSPS?  If  so,  what? 

16.  Do  you  believe  that  your  organization  will  be  better  off  under  NSPS?  Why  or  why  not? 
What  makes  you  think  this? 

■  What  do  you  think  the  long  term  effects  of  NSPS  will  be? 

17.  What  is  the  one  thing  about  NSPS  you  would  most  like  to  change? 

Do  you  have  anything  to  add? 

Remember,  if  you  have  issues  you  would  like  to  bring  up,  but  are  not  comfortable  doing  so  in 
this  group,  or  if  you  think  of  any  other  comments  you  would  like  to  make,  you  can  contact  me 
at _ to  set  up  a  telephone  interview. 

Thanks  for  meeting  with  us  today. 

Group  Interview  Protocol  -  Managers/Pav  Pool  Panel 

Thank  you  for  joining  us  today.  My  name  is _ from _ and  I  will  facilitate  this 

group  interview.  I  am  joined  by 

[Ask  the  visiting  group  to  introduce  themselves  and  to  state  their  organization] 

As  you  know,  Spiral  1  of  the  National  Security  Personnel  System  (NSPS)  covers  more  than 
100,000  employees,  and  another  70,000  are  converting  this  year  in  Spiral  2.  We  are  in  the 
process  of  conducting  focus  groups  and  interviews  at  12  sites  across  the  country  to  help 
evaluate  NSPS  as  it  was  implemented  in  Spiral  1. 

Although _ will  be  taking  detailed  notes  on  the  information  collected  in  this  focus 

group,  all  responses  will  remain  anonymous  and  non-attributional.  That  is,  we  will  not  link  your 
name  to  any  of  the  comments  that  you  make.  The  results  of  this  session  will  be  combined  with 
other  data  to  evaluate  NSPS. 

This  session  is  expected  to  last  90  minutes.  We're  interested  in  getting  your  perspective  both  as 
senior  managers  who  deal  with  all  aspects  of  NSPS,  and  as  pay  pool  officials.  We  will  begin  by 
asking  about  your  management  experiences  with  pay,  staffing,  and  pay  banding  aspects  of 
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NSPS,  and  then  move  on  to  pay  pool  panel  questions.  Before  we  begin,  I'd  like  for  you  to 
introduce  yourselves.  Please  give  us  your  name,  which  organization  you're  with,  and  how  long 
you've  been  here  at _ . 

Thanks  to  all  of  you  for  making  time  to  be  with  us  today. 

Are  there  any  questions? 

Let's  start  off  with  the  some  questions  related  to  staffing  your  workforce. 

1.  How  do  you  think  NSPS  is  affecting  your  ability  to  recruit  and  retain  high  quality 
employees? 

■  Are  you  able  to  keep  pay  reasonably  competitive? 

■  How  does  your  component  or  local  pay  policy  affect  your  pay  offers  and  decisions? 

■  What  kinds  of  effects  does  pay  banding  have  on  local  reassignment  and  work 
assignment  processes? 

2.  Are  you  able  to  strike  a  balance  between  the  pay  expectations  of  current  employees  and 
the  pay  expectations  of  new  hires? 

Let's  move  on  to  the  pay  pools,  starting  with  some  questions  about  planning  for  the  panel. 

3.  How  did  you  decide  who'd  be  members  of  the  pay  pool  panels,  and  whether  or  not  to 
use  subpool  panels? 

4.  Please  tell  us  about  the  objectives,  principles,  and  business  rules  you  established  for  the 
pay  pool  -  why  you  decided  on  them  and  how  they  worked  out. 

■  Did  you  have  to  work  out  any  issues  with  the  pay  pool  process  along  the  way? 

■  How  did  you  and  the  panel  members  prepare  yourselves  for  the  panel's  work? 

■  How  did  your  deliberation  process  work?  Was  it  effective? 

■  What  about  business  rules  for  EPI?  OAR?  Proration?  Control  points? 

5.  What  feedback  or  information  from  you  mock  pay  pool  experience  did  you 
communicate  to  employees  and  supervisors? 

■  Did  the  panel  give  rating  officials  (supervisors)  suggestions  on  ways  to  improve 
employees'  performance  objectives  or  assessment  input? 

■  Did  the  panel  share  general  results  of  mock  ratings? 

■  Did  they  panel  mainly  use  the  experience  internally,  to  test  and  improve  its 
processes? 

Next,  we  would  like  to  hear  your  opinion  about  the  rating  and  performance  payout  processes. 

6.  What  did  you  and  the  panel  do  when  panel  members  disagreed  with  a  rating  official's 
recommended  rating? 

■  If  there  were  more  than  minor  disagreements,  what  kinds  of  issues  were  there? 

■  How  did  you  resolve  disagreements  or  get  extra  information? 

■  What  sort  of  feedback  to  you  give  rating  officials  when  you  changed  a  rating  from 
their  recommendation? 

■  How  well  did  the  final  employee  ratings  in  your  pay  pool  conform  to  your  shared 
performance  rating  expectations? 

7.  Describe  the  review  process  your  panel(s)  used  to  achieve  rating  consistency  and  equity, 
for  example,  among  similar  jobs  or  between  jobs  in  different  pay  bands. 

■  How  well  did  your  process  for  building  a  shared  understanding  of  performance 
ratings  work? 
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■  Did  you  review  everything  for  all  employees  or  did  you  review  selectively? 

■  How  well  did  this  work? 

■  How  effective  was  the  process  of  determining  final  shares  and  payout  distributions 
between  salary  and  bonus? 

■  Would  you  do  anything  differently  next  time? 

8.  How  useful  were  the  performance  plans  and  assessment  input?  The  standard 
benchmarks  and  performance  indicators? 

■  How  about  the  Compensation  Workbench  (CWB),  Pay  Pool  Analysis  Tool,  or  other 
tools  you  used? 

9.  What  kinds  of  information  did  you  communicate  to  the  rating  officials? 

■  How  effective  was  this? 

■  Would  you  do  anything  differently  next  time? 

10.  How  did  you  decide  what  pay  pool  and  payout  results  to  present  to  the  workforce? 

■  How  could  fairness  and  credibility  of  the  rating  process  be  enhanced? 

11.  How  effective  was  the  rating  reconsideration  process?  To  what  degree  was  the  rating 
reconsideration  process  effective? 

■  How  much  use  did  employees  make  of  it? 

■  What  percentage  of  ratings  was  changed? 

■  Do  you  have  a  sense  of  whether  employees  generally  were  satisfied  that  they  got  a 
fair  opportunity  to  make  their  case? 

12.  Was  there  enough  time  to  do  the  pay  pool  activities  you  felt  should  be  done? 

■  What  took  more  time  than  you  expected? 

■  Do  you  think  it  will  take  a  similar  amount  of  time  in  the  future? 

■  What  shortcuts  or  alternative  approaches  did  you  use  to  save  time? 

13.  What  was  hardest  for  your  panel?  Why? 

14.  What  changes  are  you  planning  to  make  based  on  your  pay  pool  panel  experience  this 
past  cycle? 

We  have  some  final,  general  questions. 

15.  Overall,  do  you  believe  that  high  performers  will  be  better  rewarded  under  NSPS  than 
under  the  previous  system?  Why  or  why  not? 

■  How  do  you  think  pay  progression  and  rewards  under  NSPS  compare  to  what  would 
have  happened  under  the  previous  system? 

16.  Do  you  believe  that  your  organization  will  be  better  off  under  NSPS?  Why  or  why  not? 
What  makes  you  think  this? 

■  Where  do  you  think  NSPS  will  be  most  helpful  to  your  organization  ? 

■  What  do  you  like  most  about  NSPS?  Least? 

■  What  is  the  one  thing  about  NSPS  you  would  most  like  to  change? 

Is  there  anything  you  would  like  to  add? 

Thanks  again  for  taking  time  with  us  today. 

Group  Interview  Protocol  -  Implementation  Team 

Thank  you  for  joining  us  today.  My  name  is _ from _ and  I  will  facilitate  this 

group  interview.  I  am  joined  by 

[Ask  the  visiting  group  to  introduce  themselves  and  to  state  their  organization] 
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As  you  know.  Spiral  1  of  the  National  Security  Personnel  System  (NSPS)  has  been  implemented 
for  over  100,000  DoD  employees.  We  are  in  the  process  of  conducting  focus  groups  and 
interviews  as  part  of  12  site  visits  across  the  country  to  help  evaluate  NSPS  Spiral  1. 

Although _ will  be  taking  detailed  notes  on  the  information  collected  in  this  focus 

group,  all  responses  will  remain  anonymous  and  non-attributional.  That  is,  we  will  not  link  your 
name  to  any  of  the  comments  that  you  make.  The  results  of  this  session  will  be  combined  with 
other  data  to  evaluate  NSPS. 

Because  the  information  we  are  collecting  in  this  session  is  so  critical  and  the  time  for  this 
session  is  limited,  we  ask  that  you  please: 

■  Speak  clearly 

■  Speak  one  at  a  time 

■  Avoid  side  conversations 

This  session  is  expected  to  last  for  up  to  two  hours.  We  will  ask  you  about  your  experiences  as 
members  of  the  implementation  team.  Before  we  begin,  I'd  like  for  you  to  introduce 
yourselves.  Please  give  us  your  name,  which  organization  you're  with,  how  long  you've  been 
here  at _ ,  and  your  role  on  the  implementation  team. 

Thanks  to  all  of  you  for  making  time  to  be  with  us  here  today. 

Are  there  any  questions? 

Would  you  tell  us  a  little  about  your  organization  and  the  staff  that's  covered  by  NSPS? 

Let's  begin  with  a  discussion  of  how  your  effort  started: 

1.  What  kinds  of  guidance  and  authority  did  you  receive  from  the  chain  of  command 
related  to  preparing  for  and  managing  the  transition  to  NSPS? 

■  How  valuable  was  it? 

2.  What  kinds  of  training  or  other  preparation  and  information  did  you  receive,  to  help  you 
with  planning  and  carrying  out  NSPS  implementation: 

■  How  valuable  was  it? 

■  Was  there  anything  extra  that  you  think  that  implementation  team  members  need  to 
prepare  them,  that  you  didn't  have? 

3.  Tell  us  about  the  kinds  of  communication,  outreach,  and  change  management  activities 
you  conducted.  [Examples:  informal  information  sessions,  town  hall  meetings, 
newsletters,  presentations  about  the  organization's  strategic  plan] 

■  How  successful  were  these  efforts? 

m  Would  you  do  anything  differently  if  you  had  the  chance? 

4.  What  training  did  your  team  arrange  for  employees,  supervisors,  and  other  officials? 

■  What  did  you  do  to  make  sure  that  everyone  who  was  supposed  to  be  trained 
attended  the  training  or  used  the  training  resources? 

■  What  kinds  of  reinforcing  or  supplemental  training  did  you  provide? 

■  What  sorts  of  adjustments  did  you  make  to  the  training  based  on  course  feedback? 

■  What  would  you  do  differently  to  train  and  prepare  the  workforce  for  a  change  like 
NSPS,  based  on  your  experience? 

Please  tell  us  about  how  your  team  was  structured  and  the  resources  you  used. 

5.  Was  the  implementation  team  sized  right  for  your  organization? 

■  What  was  the  team's  composition?  For  example,  did  it  include: 
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•  Both  civilian  and  military  members? 

•  Both  supervisory  and  non-supervisory  employees? 

•  Individuals  with  pay-for-performance  experience? 

•  Individuals  with  specific  knowledge,  such  as  finance,  HR,  legal,  public  affairs, 

information  technology,  or  change  management? 

■  How  effective  was  the  team?  What  would  you  change  about  it? 

■  What  kinds  of  resources  were  available  to  the  team? 

•  Were  team  members  dedicated  full-time? 

•  Budget? 

•  Data/IT? 

•  Trainers? 

•  Other  resources? 

•  Did  your  team  get  good  use  out  of  the  Readiness  Tool? 

6.  Were  your  resources  sufficient  to  accomplish  the  team's  goals? 

■  If  not,  what  other  resources  did  you  need? 

7.  What  were  your  team's  biggest: 

■  Successes? 

■  Disagreements? 

■  Failures? 

To  conclude  the  session,  we  would  like  to  ask  you  a  few  general  questions. 

8.  Did  you  allow  enough  time  as  the  implementation  team  to  do  all  the  implementation 
activities  you  felt  were  necessary? 

9.  Do  you  believe  that  your  team  was  successful  in  helping  the  workforce  transition  to 
NSPS?  Why  or  why  not? 

10.  What  lessons  would  you  like  to  pass  along  from  your  implementation  experience? 

Is  there  anything  else  you  would  like  to  add? 

Thank  you  for  meeting  with  us  today. 

Interview  Protocol  -  Commander/Director 

NOTE:  This  protocol  should  be  combined  with  the  PRA  protocol  when  the  Commander  is  also 
the  PRA. 

Thank  you  for  meeting  with  us  today.  We  are  one  of  three  teams  conducting  site  visits  to  Spiral 
1  organizations  as  part  of  a  comprehensive  NSPS  program  evaluation.  Please  let  us  introduce 
ourselves.  My  name  is  _ from  _ and  I  am  joined  by 

Describe  the  site  visit  purpose  and  agenda.  Identify  others  with  whom  the  team  will  be 
meeting. 

Before  we  begin,  are  there  any  questions? 

Let's  begin  with  some  questions  about  your  role. 

1.  What  is  your  role  in  helping  your  employees  and  supervisors  make  the  transition  to 
NSPS? 
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2.  [Ask  this  question  only  if  the  Commander  is  at  the  Command  or  Agency  level]  What 
Command  (or  Agency)  guidance  and  assistance  has  been  given  to  subordinate  units 
about  their  responsibilities  and  authority  under  NSPS? 

3.  How  does  NSPS  affect  the  way  you  run  your  organization? 

■  Please  describe  any  internal  oversight  or  control  processes  you've  put  into  effect  to 
manage  risks  under  NSPS,  like  equitable  ratings  and  salary  cost  growth. 

4.  How  does  NSPS  affect  your  ability  to  staff  your  workforce  and  meet  your  mission? 

■  Are  you  satisfied  with  the  flexibility  and  local  discretion  you  have  under  NSPS,  or  are 
there  aspects  where  you'd  like  a  freer  hand? 

■  What  changes  are  you  starting  to  see  in  organizational  performance  that  you  can 
relate  to  NSPS? 

5.  Over  the  long  term,  how  worthwhile  is  the  investment  of  time  and  effort  you  and  your 
workforce  are  putting  into  performance  management?  (e.g.,  training,  performance 
planning,  discussions,  assessment,  and  panel  deliberations) 

Finally,  we  have  some  general  questions. 

6.  What  feedback  have  you  received  regarding  NSPS  from  the  supervisors  and  employees 
in  your  organization? 

7.  What  is  the  one  thing  about  NSPS  you  would  most  like  to  change,  and  why? 

8.  Do  you  believe  that  your  organization  will  be  better  off  under  NSPS?  Why  or  why  not? 
(What  makes  you  think  this?) 

■  What  do  you  think  the  long  term  effects  of  NSPS  will  be? 

Is  there  anything  you  would  like  to  add?  Thank  you  for  taking  time  with  us  today. 

Interview  Protocol  -  Performance  Review  Authority 

NOTE:  This  protocol  should  be  combined  with  the  Commander  protocol  when  the 
Commander  is  also  the  PRA. 

Thank  you  for  meeting  with  us  today.  We  are  one  of  three  teams  conducting  site  visits  to  Spiral 
1  organizations  as  part  of  a  comprehensive  NSPS  program  evaluation.  Please  let  us  introduce 
ourselves.  My  name  is _ from _ and  I  am  joined  by _ . 

Describe  the  site  visit  purpose  and  agenda.  Identify  others  with  whom  the  team  will  be 
meeting. 

Although _ will  be  taking  careful  notes,  all  of  your  responses  will  remain  anonymous. 

The  results  of  this  session  will  be  combined  with  other  interviews  here  and  eventually  at  other 
organizations  as  we  evaluate  NSPS. 

Before  we  begin,  do  you  have  any  questions? 

Let's  begin. 

1.  What  are  your  key  interests  and  objectives  as  the  PRA? 

2.  How  well  did  the  pay  pool  process  work  in  your  organization? 

3.  Please  tell  us  about  the  business  rules  your  pay  pools  used,  why  you  chose  those 
approaches,  and  how  they  worked  out. 
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■  Assigning  shares? 

■  Control  points? 

■  Proration? 

m  Use  of  EPI  and  OAR? 

■  How  to  split  pay  increases  between  base  pay  and  bonus? 

4.  How  did  you  decide  on  the  number  and  structure  of  the  pay  pool(s)? 

5.  Please  describe  the  nature  of  your  interaction  with  the  pay  pool  panel(s)  before,  during, 
and  after  their  deliberations? 

■  Topics? 

■  Purpose ,  e.g.,  discuss/decide  issues,  give  direction 

6.  To  what  extent  did  you  tie  mission  results  and  outcomes  to  pay  pool  funding,  the  rating 
process,  or  other  aspects  of  the  payout,  such  as  share  options? 

7.  What  steps  have  you  taken  to  make  your  performance  rating  process  and  payouts  fair, 
equitable,  and  transparent? 

■  What  kinds  of  guidance,  if  any,  did  you  give  to  rating  officials? 

■  Did  you  have  the  information  and  charts  you  needed  to  review  pay  pool  results? 

And  can  you  tell  us  how  useful  the  pay  pool  analysis  tool  and  the  compensation 
workbench  were  in  providing  that  data? 

■  What  information  did  you  arrange  to  release  to  the  workforce  about  business  rules 
and  your  organization's  pay  pool  outcomes?  Did  you  have  any  reservations  about 
releasing  the  information? 

■  Do  you  intend  to  make  -  or  "recommend,"  if  PRA  is  departing  -  any  changes  to  pay 
pool  policies  and  practices? 

8.  How  did  the  rating  reconsideration  appeal  process  work  out? 

To  conclude,  we  would  like  to  ask  you  a  few  general  questions  about  NSPS. 

9.  What  do  you  like  most  about  NSPS?  Least? 

■  What  is  the  one  thing  about  NSPS  you  would  most  like  to  change? 

10.  Do  you  believe  that  your  organization  will  be  better  off  under  NSPS?  What  makes  you 
think  this? 

■  What  do  you  think  the  long  term  effects  of  NSPS  will  be? 

Is  there  anything  you'd  like  to  add? 

Thank  you  for  meeting  with  us  today. 
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The  study  team  conducted  58  focus  groups  as  part  of  site  visits  to  12  NSPS  Spiral  1 
organizations.  Participants  for  the  focus  groups  were  randomly  selected  and  attended 
voluntarily.  Each  participant  was  asked  to  complete  a  short  questionnaire  requesting 
demographic  and  background  information,  and  most  completed  it.  This  appendix  provides 
results  from  these  participant  questionnaires.  Section  E.l  summarizes  the  demographic 
characteristics  of  the  participants.  Sections  E.2  and  E.3  report  demographics  by  pay  band  and 
pay  schedule  and  describe  participants'  years  of  experience  as  DoD  civilians. 

Demographics  of  Focus  Group  Participants 

A  total  of  635  participants  completed  demographic  questionnaires.  The  total  number  of 
participants  at  a  site  ranged  from  a  low  of  21  to  a  high  of  75.  Non-supervisory  employees  were 
the  largest  group,  comprising  60.2%  of  the  total.  Table  E.l  provides  employee  and  supervisor 
counts  and  percentages. 


Table  E.l-  Supervisory  Status  of  Focus  Groups  Participants 


Type  of  Focus  Group 

Number  of 
Participants 

Percentage  of 
Total  Sample 

Supervisor 

253 

39.8 

Non-Supervisory  Employee 

382 

60.2 

Total 

635 

100.0 

Table  E.2  presents  the  gender  composition  of  the  focus  groups.  Males  were  about  54%  of  all 
participants. 


Table  E.2  -  Gender  of  Focus  Groups  Participants 


Gender 

Number  of 

Percentage  of 

Participants 

Total  Sample 

Male 

341 

53.7 

Female 

292 

46.0 

Missing 

2 

0.3 

Total 

635 

100.0 

As  shown  in  Table  E.3,  focus  group  participants  represented  all  race  groups.  Whites  were  the 
largest  group  at  just  over  70%,  followed  by  Blacks  or  African  Americans  and  Asians,  Native 
Americans  or  Alaska  Natives,  and  Native  Hawaiians  or  Other  Pacific  Islanders  each  comprised 
less  than  1%  of  the  participants.  Table  E.4  illustrates  that  less  than  5%  of  the  participants 
indicated  Hispanic  or  Latino  origin. 


Table  E.3  -  Race/National  Origin  of  Focus  Groups  Participants 


Race/National  Origin 

Number  of 
Participants 

Percentage  of 
Total  Sample 

White 

455 

71.7 

Black  or  African  American 

75 

11.8 

Asian 

38 

6.0 

Native  American  or  Alaska  Native 

4 

0.6 

Native  Hawaiian  or  Other  Pacific  Islander 

3 

0.5 

Other 

16 

2.5 

Missing 

44 

6.9 

Total 

635 

100.0 

May  15,  2009 


E-l 


Appendix  E:  Focus  Group  Porticipont  Demographics 


NS  PS  Spiral  1  Evaluation  Report 


Table  E.4  -  Hispanic  or  Latino  Origin  of  Focus  Groups  Participants 


Hispanic  or  Latino 
Origin 

Number  of 
Participants 

Percentage  of  All 
Participants 

Yes 

30 

4.7 

No 

447 

70.4 

Missing 

158 

24.9 

Total 

635 

100.0 

Focus  Group  Participants  bv  Pay  Band  and  Pay  Schedule 

As  shown  in  Table  E.5,  focus  group  participants  represented  eleven  NSPS  pay  schedules  and  all 
four  pay  bands.  In  addition,  some  supervisors  were  active  duty  military.  The  largest  group  of 
participants  was  Professional/Analytical,  Pay  Band  2  (YA-2),  followed  closely  by 

Supervisor/Manager,  Pay  Band  2  (YC-2). 


Table  E.5  —  Pay  Band  and  Pay  Schedule  of  Focus  Groups  Participants 


Pay  Schedule 

Pay  Band 

Total  Count 

1 

2 

3 

4 

Missing 

Standard  Career  Group 

Professional/Analytical  (YA) 

17 

141 

42 

0 

3 

200 

Technician/Support  (YB) 

30 

19 

2 

0 

5 

51 

Supervisor/Manager  (YC) 

7 

107 

46 

0 

0 

160 

Student  (YP) 

4 

0 

0 

0 

0 

4 

Scientific  and  Engineering  Career  Group 

Professional  (YD) 

5 

48 

25 

0 

1 

78 

Technical/Support  (YE) 

0 

7 

4 

0 

1 

11 

Supervisor/Manager  (YF) 

0 

26 

29 

0 

0 

55 

Medical  Career  Group 

Physician/Dentist  (YG) 

0 

1 

0 

0 

0 

1 

Professional  (YH) 

0 

9 

1 

0 

2 

10 

Technician/Support  (Yl) 

0 

0 

0 

0 

0 

0 

Supervisor/Manager  (YJ) 

1 

10 

0 

1 

0 

12 

Investigative  and  Protective  Services  Career  Group 

Investigative  (YK) 

0 

0 

0 

0 

0 

0 

Fire  Protection  (YL) 

0 

0 

0 

0 

0 

0 

Police/Security  Guard  (YM) 

0 

0 

0 

0 

0 

0 

Supervisor/Manager  (YN) 

4 

2 

1 

0 

0 

7 

Other 

34 

0 

Total 

68 

370 

150 

1 

46 

635 

The  majority  (58%)  of  participants  in  the  focus  groups  had  no  prior  military  service. 

Years  as  a  DoD  Civilian 

The  average  years  of  experience  as  a  DoD  civilian  was  12.3  years.  The  overall  experience 
reported  by  participants  ranged  from  less  than  one  year  to  40  years.  The  largest  number  of 
participants  (39.6%)  reported  having  fewer  than  five  years  of  experience.  Male  participants 
averaged  11.5  years,  and  females  averaged  13.2  years.  Table  E.6  presents  years  as  a  DoD 
civilian  for  all  participants  by  gender. 
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Table  E.6  -  Years  as  a  DoD  Civilian-By  Gender 


Gender 

Mean 

Number  of  Participants 

Male 

11.5 

318 

Female 

13.2 

284 

Missing 

1 

Total 

12.3 

603 

Participants  averaged  5.4  years  in  their  current  position,  with  males  averaging  5.5  years  and 
females  5.3  years.  The  overall  range  of  experience  in  the  current  position  was  from  less  than 
one  year  to  33  years.  The  majority  of  the  participants,  57%,  reported  fewer  than  three  years. 

The  majority  of  focus  group  participants  had  no  previous  experience  in  an  NSPS  Spiral  1 
organization,  with  only  about  9%  having  worked  in  more  than  one  Spiral  1  organization. 
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While  integrating  the  overall  findings,  the  study  team  also  identified  some  of  the 
industrial/organizational  psychology  literature  that  may  be  relevant  to  understanding  the 
results  of  this  study.  The  topics  of  this  literature  review  include: 

■  Implementation  of  pay-for-performance  systems 

■  Writing  high-quality  performance  objectives 

■  Factors  that  affect  the  accuracy  of  performance  ratings 

■  Strategies  for  addressing  poor  performance 

■  Reactions  to  perceived  organizational  injustice 

■  Strategies  for  recruiting  and  retaining  employees. 

For  each  topic,  we  provide  a  brief  review  of  the  literature  with  emphases  on  (1)  how  the 
literature  relates  to  the  results  of  the  study  and  (2)  recommendations  for  improved  practice. 

Implementation  of  Pav-for-Performance  Systems 

Research  has  found  that  strengthening  the  relationship  between  pay  and  performance 
increases  job  performance  (Locke  et  al.,  1980),  and  the  use  of  pay-for-performance  (PFP) 
systems  appears  to  be  growing  (Heneman  et  al.,  2000;  as  cited  in  Rynes,  Gerhart,  &  Parks, 
2005).  A  study  entitled  "The  Rewards  of  Work,"  conducted  by  Sibson  &  Company,  found  that 
employees  who  are  satisfied  with  their  organization's  pay  system  were  more  committed  to 
their  organization  and  less  likely  to  leave  (LeBlanc  &  Mulvey,  1998).  Most  Americans  prefer 
merit-based  pay  systems  in  which  fixed  raises,  as  opposed  to  one-time  bonuses,  are  added  to 
their  base  salary.  In  addition,  Americans  prefer  pay  raises  based  on  their  individual 
performance,  and  not  based  on  their  team's  or  company's  performance.  Those  most  in  favor  of 
merit-based  pay  tend  to  be  the  organization's  highest  performers  (LeBlanc  &  Mulvey,  1988). 

PFP  is  thought  to  influence  motivation  and  performance  through  the  "incentive  effect"  and  the 
"sorting  effect"  (Rynes,  Gerhart  &  Parks,  2005).  The  "incentive  effect"  refers  to  PFP's  influence 
on  job  performance.  The  "sorting  effect"  refers  to  the  process  by  which  employees  self-select  in 
and  out  of  organizations.  An  organization's  pay  system  can  affect  the  type  of  individuals  who 
remain  with  the  organization.  Given  that  high  performers  prefer  merit-based  pay,  the  "sorting 
effect"  might  increase  organizational  effectiveness  through  the  retention  of  high  performers 
and  the  attrition  of  low  performers.  For  example,  Lazear  (1986)  found  that  organizational 
productivity  increased  significantly  as  low  performing  employees  were  replaced  by  higher 
performing  employees  over  time. 

Although  PFP  has  a  large  upside,  it  is  also  fraught  with  risk.  PFP  success  depends  on  the 
accurate  measurement  of  employee  job  performance.  Measures  of  job  performance  can  be 
objective  or  subjective,  but  each  measurement  approach  presents  unique  limitations.  Objective 
measures  of  job  performance  can  include  sales  growth,  profitability  and  share  prices. 
Unfortunately,  these  measures  are  susceptible  to  factors  outside  the  employee's  control.  For 
example,  sales  growth  is  subject  to  prevailing  economic  conditions.  Therefore,  a  salesperson's 
performance  might  more  accurately  reflect  the  economic  climate  than  the  salesperson's  actual 
job  performance.  Subjective  measures  of  job  performance,  such  as  supervisor  ratings,  can  be 
used  for  any  type  of  job  and  capture  a  broad  array  of  performance  dimensions  (Rynes,  Gerhart, 
&  Parks,  2005).  Unfortunately,  subjective  behavior-oriented  measures  contain  rater  bias  and 
poorly  differentiate  among  employees  (Murphy  &  Cleveland,  1995;  see  also  the  discussion  of 
sources  of  error  in  performance  ratings  in  this  chapter).  Furthermore,  supervisors  might  have 
an  incentive  to  rate  all  of  their  employees  equally  in  order  to  ensure  social  harmony  (Heneman 
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&  Judge,  2000;  as  cited  in  Rynes  et  al.,  2005),  especially  if  promotion  and  pay  decisions  are 
based  upon  such  ratings  (Jawahar  &  Williams,  1997). 

Effective  pay-for-performance  systems  depend  on  careful  implementation  and  sensitivity  to 
employee  and  organizational  considerations.  A  recent  study,  commissioned  by  the  Office  of  the 
Director  of  National  Intelligence  (ODNI),  identified  critical  success  factors  and  impediments  for 
successful  pay-for-performance  implementation.  The  critical  success  factors  included  system 
credibility  and  transparency,  sufficient  financial  support  for  the  program,  supervisory  and 
managerial  training  and  accountability,  employee  education  and  training,  top  management 
support,  alignment  among  employee  and  organizational  goals,  opportunity  for  employee 
feedback,  and  union  engagement  early-on.  The  study  also  identified  critical  impediments  to 
successful  pay-for-performance  implementation.  These  included  lack  of  transparency  in  the 
pay-for-performance  process,  failure  to  solicit  employee  feedback,  unfairness  perceptions, 
rating  distortions  by  supervisors,  lack  of  communication  about  the  pay-for-performance 
system,  inadequate  managerial  and  employee  training,  inability  to  deal  with  poor  performers, 
and  failure  to  engage  unions.  These  success  factors  and  impediments  are  similar  to  those 
identified  in  a  report  prepared  by  the  National  Association  for  Public  Administration. 

Writing  High-Quality  Performance  Objectives 

A  common  finding  across  organizational  levels  was  the  difficulty  in  developing  effective 
performance  objectives.  The  challenges  that  employees  and  supervisors  are  facing  with 
developing  objectives  under  NSPS  are  not  uncommon.  However,  considering  the  critical  role  of 
the  objectives  in  the  overall  performance  evaluation  process,  it  is  important  to  address  these 
issues. 

Pulakos  and  O'Leary  (2009)  identified  many  common  challenges  associated  with  developing 
performance  objectives  and  mitigation  strategies  for  addressing  these  challenges,  several  of 
which  are  directly  relevant  to  NSPS. 

Training  Managers  and  Staff  To  Write  Effective  Objectives 

NSPS  has  provided  training  to  employees  and  supervisors  on  writing  objectives.  However,  the 
quality  of  objectives  even  after  training  varies  greatly  and  many  NSPS  employees  and 
supervisors  admit  to  being  unclear  on  how  to  write  effective  objectives. 

One  strategy  to  supplement  training  is  for  trained  human  resources  staff  or  higher  level 
managers  to  review  the  objectives  at  the  beginning  of  the  performance  period.  Objectives  can 
then  be  modified  immediately,  if  necessary,  so  that  employees  have  a  clearer  understanding 
from  the  outset  as  to  the  expectations  for  their  performance  over  the  performance  period.  The 
higher  level  managerial  review  also  allows  an  assessment  of  objectives  for  employees  in  similar 
positions  (e.g.,  same  job  series,  band  level,  and  pay  schedule)  to  ensure  that  they  are  being 
assessed  on  comparable  objectives  and  that  the  objectives  for  one  employee  are  not  unduly 
easier  or  more  difficult  than  other  employees  in  similar  situations. 

Another  strategy  is  to  develop  supplementary  resources  to  assist  supervisors  and  employees  in 
developing  objectives.  For  example,  retaining  objectives  in  a  searchable  database  can  facilitate 
both  the  quality  and  consistency  of  individual  objectives.  These  can  be  used  again  verbatim  or 
refined  and  edited  over  time  to  develop  future  objectives.  In  addition,  a  supplementary 
guidebook  can  serve  as  a  step-by-step  job  aid  that  employees  and  supervisors  can  work 
through  when  developing  objectives.  Key  elements  of  this  type  of  guidebook  include  a  step-by- 
step  process  to  develop  objectives,  key  questions  to  answer  when  developing  objectives,  and 
examples  of  performance  objectives  with  corresponding  explanations  as  to  whether  the 
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examples  are  effective  or  ineffective.  For  many  supervisors  and  employees,  showing  a  variety  of 
job-relevant  examples  is  key  to  understanding  how  to  apply  the  general  principles  of 
developing  objectives  (e.g.,  SMART  framework)  to  their  work. 

Ensuring  Objectives  Are  Job  Relevant 

In  more  routine,  standard,  and  predictable  jobs,  it  is  often  possible  to  predefine  a  set  of 
objectives  that  apply  uniformly  to  all  employees  at  a  given  level  using  standard  job  analytic 
procedures.  This  not  only  saves  time  that  would  otherwise  be  spent  by  each  manager  and 
employee  developing  individual  objectives,  but  it  also  ensures  that  all  employees  in  the  same 
job  are  held  accountable  for  delivering  the  same  results.  Standardized  objectives  are  not  only 
the  most  fair  for  employees,  but  they  also  allow  straightforward  comparisons  to  be  made 
between  employees  in  terms  of  the  results  they  delivered. 

In  more  unique  jobs  and  situations,  it  may  be  impossible  to  predefine  objectives  that  apply 
across  positions,  jobs,  or  organizations.  Even  though  a  group  of  employees  may  occupy  a  given 
job,  the  specific  results  each  individual  is  expected  to  achieve  may  vary  depending  on  the 
nature  of  his  or  her  assignments.  To  the  extent  that  people  holding  similar  jobs  have  different 
goals  and  objectives,  evaluating  and  comparing  their  performance  in  a  fair  and  standardized 
manner  becomes  increasingly  challenging.  Under  these  circumstances,  one  recommendation  is 
to  develop  individual  objectives  that  further  define  critical  tasks  from  a  comprehensive  job 
analysis.  This  helps  to  ensure  that  a  common  base  of  job-relevant  information  is  used  to 
develop  objectives.  Objectives  derived  in  this  manner  will  contain  more  specific  information 
than  the  tasks  or  work  behavior  statements,  such  as  what  specific  project,  customer,  product, 
etc.  the  employee  is  responsible  for  and  what  specific  quality,  quantity,  and  timeliness  criteria 
will  be  measured. 

Setting  Objectives  in  Fluid  Situations 

Setting  specific  objectives  in  advance  may  be  extremely  difficult  for  some  jobs  (Cascio,  1998; 
Levinson,  2005).  Jobs  that  lend  themselves  best  to  setting  objectives  have  relatively  static 
performance  requirements  and  definable  productivity  metrics,  both  of  which  are  uncommon  in 
many  of  today's  jobs.  For  jobs  that  are  fluid  and  unpredictable,  or  in  situations  where 
unforeseen  circumstances  regularly  interfere  with  attaining  objectives,  it  may  be  necessary  to 
alter  or  completely  revise  an  employee's  objectives  during  the  rating  period.  Managers  and 
employees  need  to  be  prepared  to  make  changes  to  the  objectives  as  the  situation  or  priorities 
change.  Obviously,  to  the  extent  that  a  situation  is  chronically  volatile,  requirements  for 
constant  changes  to  the  formal  performance  plan  may  prove  to  be  impractically  time 
consuming.  An  alternative  strategy  for  jobs  that  are  in  flux  is  to  set  shorter-term  objectives  that 
are  more  predictable.  Feedback  can  be  given  during  the  rating  period  as  employees  meet  key 
milestones.  In  fact,  given  the  fluid  nature  of  many  work  environments  and  jobs,  some  experts 
have  argued  against  setting  longer-term  objectives  and  instead  recommend  setting  shorter- 
term  goals  as  the  work  evolves. 

Factors  That  Affect  the  Accuracy  of  Performance  Ratines 

One  key  finding  of  this  evaluation  was  that  employees  and  supervisors  are  not  always  clear  on 
what  behaviors  constitute  different  performance  levels.  For  example,  several  supervisors  and 
employees  indicated  that  they  were  unsure  of  how  employees  could  achieve  a  rating  greater 
than  3.  Additionally,  supervisors  and  employees  expressed  confusion  on  how  to  apply  the 
contributing  factors. 
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Managers  have  been  shown  to  possess  their  own  idiosyncratic  and  implicit  theories  of 
performance  -  that  is,  what  specific  performance  factors  and  evaluation  criteria  they  feel  are 
most  important  to  focus  on  in  rating  job  performance  (Borman,  1983).  Even  when  defined 
performance  standards  are  in  place,  managers  still  tend  to  use  their  own  theories  of 
performance  effectiveness  and  idiosyncratic  rating  standards  to  evaluate  employees  (Jamieson, 
1973;  Strauss,  1972).  For  example,  one  manager  may  be  impressed  by  an  employee's 
performance  of  a  relatively  simple  information  cataloguing  project,  while  another  manager  may 
feel  that  an  employee  in  the  same  job  and  level  who  managed  the  design  and  implementation 
of  a  complex  information  management  system  did  not  do  enough.  With  respect  to  contributing 
factors,  one  manager  may  think  Technical  Proficiency  is  most  important  in  performing  a  job, 
while  another  may  feel  that  Cooperation/Teamwork  is  what  matters.  If  there  is  no  mechanism 
in  place  for  addressing  these  differences,  employees  who  are  performing  at  different  levels  or 
making  very  different  contributions  could  be  evaluated  as  performing  with  equal  effectiveness 
by  their  respective  managers.  The  use  of  idiosyncratic  rating  standards  not  only  undermines  the 
perceived  accuracy  of  a  performance  measurement  process  but  can  also  rightly  be  viewed  as 
unfair,  with  a  consequential  negative  impact  on  employee  acceptance  of  the  process  (Dipboye 
&  de  Pontbraind,  1981;  Greenberg,  1986). 

Due  to  the  fact  that  a  great  deal  of  human  information  processing  occurs  automatically,  we  are 
often  unaware  of  the  category  systems  we  carry  within  ourselves  (Abelson,  1976;  Nisbett  & 
Wilson,  1977).  However,  acceptance  and  use  of  a  common  category  system  for  evaluating 
employees  can  be  instilled  in  managers  through  training  that  helps  them  come  to  consensus 
about  what  performance  areas  should  be  rated  and  what  constitutes  meeting,  falling  below,  or 
exceeding  expectations  for  a  job  (DeNisi,  Cafferty,  &  Meglino,  1984;  Feldman,  1986). 

During  training,  managers  discuss  the  performance  areas  on  which  employees  will  be 
evaluated,  along  with  performance  examples  that  they  believe  exemplify  different  levels  of 
effectiveness.  This  is  invariably  an  illuminating  discussion,  because  it  is  where  managers'  own 
implicit  theories  of  performance  come  to  light,  as  they  realize  how  differently  they  might 
evaluate  a  given  example  of  performance  compared  to  other  managers.  This  approach  to 
training  has  been  coined  frame-of-reference  training  (McIntyre,  Smith,  &  Hassett,  1984), 
because  it  helps  managers  develop  and  use  common  evaluative  standards  in  rating 
performance.  Research  has  shown  that  this  type  of  training  is  effective  for  increasing 
consistency  in  the  standards  used  across  managers  as  well  as  the  accuracy  of  their  ratings 
(Pulakos,  1984,  1986). 

During  the  rating  period,  managers  should  continue  discussing  examples  of  employee 
performance  as  they  occur  to  reinforce  the  development  of  common  rating  standards.  At  the 
end  of  the  rating  cycle,  it  is  best  practice  for  managers  in  a  given  unit  to  discuss  their  ratings  of 
all  employees  and  decide  on  the  ranking  of  employees  for  decision-making  purposes.  While  this 
can  be  a  time-consuming  process  initially,  its  advantage  is  that  it  helps  to  ensure  accurate 
distinctions  between  exceptional,  successful,  and  less  effective  performers.  It  also  results  in 
more  systematic,  consistent,  and  fair  decisions  across  managers.  Importantly,  this  process  helps 
to  solidify  the  use  of  common  rating  standards  among  managers,  yielding  more  effective  and 
fair  outcomes  for  employees. 

Strategies  for  Addressing  Poor  Performance 

Another  key  finding  was  that  many  employees  and  supervisors  do  not  believe  that  appropriate 
steps  are  taken  to  deal  with  poor  performers.  This  is  not  an  uncommon  finding  in  the  public 
sector.  In  fact,  the  item,  "In  my  work  unit,  steps  are  taken  to  deal  with  a  poor  performer  who 
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cannot  or  will  not  improve"  is  continuously  among  the  lowest  rated  items  on  the  Federal 
Human  Capital  Survey  (OPM,  2004). 

One  important  way  to  handle  poor  performance  is  to  provide  timely  feedback.  Performance 
should  be  discussed  and  feedback  provided  on  an  ongoing  basis  throughout  the  rating  period 
(Kirkland  &  Manoogian,  2007).  Most  employees  want  to  do  a  good  job  and  are  appreciative  of 
ideas  for  improving  their  work  products  or  processes,  when  feedback  is  delivered  in  an 
appropriate  and  helpful  manner.  This  does  not  happen  to  the  extent  that  it  should,  however, 
because  many  managers  do  not  know  how  to  deliver  feedback  productively  and  in  a  manner 
that  will  avoid  defensive  reactions  from  employees. 

Research  has  shown  that  for  feedback  to  have  the  maximum  impact,  managers  need  to 
recognize  both  effective  and  ineffective  performance  when  it  occurs  (Wexley,  1986).  It  does  not 
help  employees  to  receive  feedback  months  after  something  happened.  Further,  by  recognizing 
and  showing  appreciation  for  effective  performance,  employees  are  more  open  to  suggestions 
for  improvement  when  these  are  needed.  The  balance  of  recognizing  both  effective  and  less 
effective  performance  demonstrates  fairness  on  the  part  of  the  manager  and  increases  the 
credibility  of  the  feedback. 

Many  managers  are  reluctant  to  give  low  ratings  that  will  result  in  negative  outcomes  for  their 
employees.  In  addition,  providing  feedback  to  a  poor  performer  can  be  very  difficult.  Many 
managers  also  do  not  want  to  have  to  justify  or  explain  why  they  gave  lower  ratings  to  an 
employee  who  may  confront  or  challenge  them,  so  the  path  of  least  resistance  is  simply  to  rate 
everyone  effectively.  What  managers  do  not  always  consider  is  that  it  is  just  as  difficult  to 
justify  to  an  effective  performer  why  he  or  she  received  the  same  outcomes  and  rewards  as  an 
obviously  less  effective  performer,  not  to  mention  the  unfairness  of  this. 

Consequently,  managers  should  be  trained  on  how  to  provide  feedback  to  all  employees, 
including  employees  who  are  not  meeting  expectations.  This  training  should  include  guidance 
that  research  has  found  to  be  important  for  feedback  to  be  effective  (Wexley,  1986; 
Cederblom,  1982;  and  Cawley,  Keeping,  &  Levy,  1998),  including: 

■  Intentionally  engage  in  ongoing,  informal  performance  conversations  regularly 

■  Provide  recognition  and  thanks  as  well  as  constructive  suggestions  for  improvement  in 
close  proximity  to  the  performance  event 

■  Ask  for  the  employee's  view  about  what  could  have  been  done  differently 

■  Be  specific  about  what  was  effective  or  ineffective 

■  Focus  on  what  the  person  did  or  did  not  do,  not  personal  characteristics 

■  Collaboratively  plan  steps  to  address  improvement  needs 

■  Offer  help  in  addressing  development  needs  and  providing  resources 

In  addition,  this  training  should  include  guidance  on  having  difficult  performance  conversations 
and  policy  for  handling  poor  performers  (e.g.,  documentation  requirements,  performance 
improvement  plan  procedures).  It  is  important  for  managers  to  know  these  policies  at  the 
beginning  of  the  performance  period  so  that  they  can  follow  the  appropriate  policies  in  a  timely 
manner  rather  than  getting  to  the  end  of  the  performance  period  and  finding  out  that  they  did 
not  follow  the  appropriate  steps  to  warrant  issuing  a  low  rating  to  an  employee. 
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Reactions  to  Perceived  Organizational  Injustice 

Organizational  justice,  psychological  contracts  and  organizational  citizenship  behaviors  are 
three  interrelated  phenomena  that  have  important,  and  sometimes  drastic,  influence  on  the 
workplace.  They  help  explain  employee  attitudinal  and  behavioral  reactions  to  organizational 
policies. 

Organizational  Justice 

Organizational  justice  refers  to  an  employee's  perception  that  the  employing  organization 
treats  him  or  her  fairly  and  respectfully.  Justice  perceptions  may  pertain  to  the  allocation  of 
resources  and  rewards  (distributive  justice,  i.e.,  who  gets  what),  the  process  by  which  resources 
and  rewards  are  allocated  (procedural  justice,  i.e.,  how  the  decisions  are  made),  and  the  way  an 
employee  is  treated  by  supervisors  and  coworkers  (interpersonal  justice).  Justice  perceptions 
have  been  linked  to  numerous  affective  and  behavioral  outcomes  (Colquitt  et  al.,  2001).  For 
example,  when  employees  experience  perceived  injustice,  they  are  more  likely  to  lower  their 
job  performance;  to  engage  in  theft,  sabotage  or  violence;  to  experience  more  conflict  with 
coworkers;  to  reduce  their  organizational  citizenship  behaviors  (OCBs)  (i.e.,  discretionary 
behaviors  such  as  helping  and  cooperating  aimed  at  facilitating  a  constructive  work 
environment);  and  to  have  negative  work  attitudes  (Weiss,  Suckow,  &  Cropanzano,  1999; 
Cropanzano,  Goldman,  &  Folger,  2003;  Greenberg,  1990).  The  effects  of  these  behavioral 
changes  are  not  always  immediately  apparent.  For  example,  an  employee  who  does  not  receive 
a  promised  raise  might  not  truly  experience  the  consequences  of  the  injustice  until  he  or  she  is 
faced  with  unexpected  financial  difficulties.  Thus,  justice  unfolds  over  time,  which  can  cause 
employees  to  reevaluate  their  fairness  perceptions  (Bobocel,  McCline,  &  Folger,  1997;  as  cited 
in  Ambrose  &  Cropanzano,  2003).  Unfairness  perceptions  easily  lead  to  a  negative  emotional 
state,  which  in  turn  is  negatively  related  to  job  performance  (Wright,  Cropanzano,  &  Meyer, 
2004).  Furthermore,  injustice  can  have  harmful  long-term  effects  on  the  employee-employer 
relationship  (Ambrose  &  Cropanzano,  2003),  culminating  in  the  employee  leaving  the 
organization.  Conversely,  increased  fairness  perceptions  are  associated  with  higher 
commitment  to  the  organization  (Colquitt  et  al.,  2001).  In  turn,  commitment  is  negatively 
related  to  turnover  intentions  and  positively  related  to  job  performance  and  OCBs  (Cropanzano, 
Rupp,  &  Byre,  2003). 

The  Psychological  Contract 

Organizational  justice  can  also  be  viewed  from  the  perspective  of  the  psychological  contract. 
The  psychological  contract  represents  the  implicit  obligations  held  by  both  parties  in  an 
employer-employee  relationship  (Rousseau  &  Tijoriwala,  1998).  Two  commonly  identified  types 
of  psychological  contracts  are  relational  and  transactional  (Rousseau,  1989).  A  relational 
psychological  contract  indicates  investment  in  a  longer-term  relationship  between  the 
organization  and  employee  and  implies  greater  opportunities  for  personal  growth  and 
promotion  within  the  organization.  In  contrast,  a  transactional  psychological  contract  is 
characterized  by  the  exchange  of  material  economic  benefits  over  a  shorter  time  horizon. 

When  an  employee  believes  that  the  organization  has  failed  to  fulfill  its  obligations,  the 
psychological  contract  between  the  employer  and  the  employee  has  been  breached  (Rousseau, 
1995).  Psychological  contract  breach  is  associated  with  numerous  negative  attitudinal  and 
behavioral  outcomes.  Attitudinal  outcomes  of  psychological  contract  breach  include  lower  job 
satisfaction,  lower  trust,  increased  turnover  intentions,  and  decreased  affective  organizational 
commitment  (  Zhao,  Wayne,  Glibkowski,  &  Bravo,  2007).  Behavioral  outcomes  of  psychological 
contract  breach  include  decreased  organizational  citizenship  behaviors  and  decreased  in-role 
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performance  (Zhao  et  al.,  2007).  Furthermore,  psychological  contracts  tend  to  become  more 
transactional  if  the  employee  perceives  that  the  employer  routinely  fails  to  fulfill  its  obligations. 

Task  and  Citizenship  Performance 

Borman  and  Motowidlo  (1993)  suggested  that  job  performance  be  conceptualized  as  two 
dimensions  of  performance;  task  and  contextual  performance.  Contextual  performance, 
commonly  referred  to  as  citizenship  performance  (Coleman  &  Borman,  2000),  refers  to  those 
behaviors  that  support  the  environment  in  which  the  organization's  technical  core  occurs.  For 
example,  an  Internet  service  provider's  technical  core  involves  providing  Internet  to  its 
customers  whereas  citizenship  performance  behaviors,  such  as  helping  and  cooperating, 
facilitate  a  harmonious  context  in  which  the  core  technical  work  occurs.  Coleman  and  Borman 
(2000)  defined  task  performance  as  an  employee's  effectiveness  in  executing  functions  of  the 
organization's  technical  core  or  providing  the  organization  with  the  materials  and  resources 
necessary  for  executing  its  core  functions.  Justice  perceptions  are  consistently  linked  to  the  task 
and  citizenship  dimensions  of  job  performance.  Underlining  the  importance  of  citizenship 
performance,  research  indicates  that  supervisors  weigh  task  and  citizenship  performance  fairly 
evenly  when  conducting  performance  evaluations  (Borman  &  Motowidlo,  1997;  Motowidlo  & 
Van  Scotter,  1994). 

The  literature  remains  conflicted  about  the  extent  to  which  task  and  contextual  performance 
are  independent  of  one  another.  There  is  also  considerable  debate  about  the  extent  to  which 
citizenship  performance  is  multi-dimensional,  as  opposed  to  reflecting  one  underlying  factor. 
Morgeson,  Reider  and  Campion  (2005)  found  that  task  and  contextual  performance  are  highly 
correlated,  suggesting  that  they  are  interdependent,  perhaps  due  to  the  highly  interdependent 
task  used  in  their  study.  LePine,  Erez  and  Johnson's  (2002)  meta-analysis  found  that  the 
dimensions  of  citizenship  performance  (e.g.,  altruism,  courtesy,  sportsmanship,  civic  virtue  and 
conscientiousness)  were  highly  intercorrelated. 

Strategies  for  Recruiting  and  Retaining  employees 

Today's  organizations  face  the  challenge  of  recruiting  and  retaining  employees  in  a  dynamic 
and  unpredictable  economic  climate.  Job  stability  has  become  uncertain,  employees  frequently 
switch  jobs  in  order  to  advance  their  careers,  and  traditional  career  systems  that  presume 
lifetime  employment  at  a  single  organization  might  be  obsolete  (Murphy,  1999).  The  imperative 
to  retain  employees  exists  because  high  turnover  rates  are  associated  with  lower  customer 
dissatisfaction  (Koys,  2001),  profitability  (Glebbeek  &  Bax,  2004)  and  productivity  (Huselid, 
1995).  Attracting  and  retaining  the  best  employees  is  therefore  especially  crucial. 

The  selection  process  is  an  important  recruitment  component.  Adverse  reactions  to  the 
selection  process  can  reduce  the  attractiveness  of  the  organization  to  the  applicant 
(Hausknecht,  Day,  &  Thomas,  2004).  Furthermore,  applicants  who  react  negatively  to  the 
selection  process  might  dissuade  others  from  applying  (Smither  et  al.,  1993).  Web-based 
recruitment  has  become  increasingly  popular  (Lievens,  van  Dam  &  Anderson,  2002)  because  it 
reduces  recruiting  costs  (Buckley,  Minette,  Joy,  &  Michaels,  2004)  at  the  same  time  that  it 
increases  the  volume  of  qualified  applicants  (Chapman  &  Webster,  2003).  Web-based 
recruitment  systems  influence  applicant  attraction  to  the  organization  (Hu,  Su,  &  Chen,  2006), 
and  are  a  valuable  recruiting  tool  if  used  properly. 

Successful  recruitment  and  retention  depend  on  an  employee's  recognition  that  the  job  is 
desirable  and  worth  keeping.  Person-environment  fit  encompasses  many  factors  that  impinge 
upon  an  employee's  assessment  of  job  attractiveness.  Person-environment  fit  is  "the 
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compatibility  between  an  individual  and  a  work  environment  that  occurs  when  their 
characteristics  are  well  matched"  (Kristof-Brown,  Zimmerman,  &  Johnson,  2005).  It  can  be 
conceptualized  as  an  employee's  fit  with  the  job  or  vocation,  supervisor,  work  group  or 
organization,  and  can  be  based  on  the  alignment  between  the  organization  and  an  individual's 
goals  (Kristof-Brown  &  Stevens,  2001),  personality  (Resick,  Baltes,  &  Shantz,  2007;  Zimmerman, 
2008),  values  (Adkins,  Ravlin,  &  Meglino,  1996)  and  workplace  rhythms  (Kristof-Brown  et  al., 
2001). 

When  person-organization  fit  is  good,  employees  are  usually  committed  to  their  employer, 
satisfied  with  their  jobs  and  do  not  intend  to  turnover  (Kristof-Brown  et  al.).  Person- 
organization  fit  is  weakly  related  to  overall  job  performance,  but  it  is  more  strongly  related  to 
the  sub-dimension  of  job  performance  called  organizational  citizenship  performance,  which 
refers  to  behaviors  such  as  helping  coworkers  or  putting  forth  extra  effort  (Kristof-Brown  et  al., 
2001).  Person-organization  fit  based  on  personality  and  values  leads  to  higher  job  satisfaction, 
which,  in  turn,  increases  employee  intentions  to  remain  with  the  organization  (Westerman  & 
Cyr,  2004). 

Employees  are  increasingly  working  in  teams  to  accomplish  organizational  objectives.  Teams 
contain  a  constellation  of  members  with  varying  personalities  and  skills,  which  yields  a  unique 
team  dynamic.  An  employee's  assessment  of  his  or  her  fit  with  the  team  tends  to  be  stable,  but 
an  employee's  team  role  might  evolve  over  time.  As  the  quality  of  fit  between  an  employee  and 
his  or  her  team  role  increases,  employees  perform  better  and  perceive  greater  personal  growth 
and  development  (DeRue  &  Morgeson,  2007). 
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Introduction 

The  National  Security  Personnel  System  (NSPS)  Program  Executive  Office  (PEO)  provided  the 
information  that  follows  from  a  supplemental  report  assessing  NSPS  automation  infrastructure. 
This  program  evaluation  report  references  such  automated  tools  as  the  Performance  Appraisal 
Application  (PAA),  Compensation  Workbench  (CWB),  and  the  NSPS  Readiness  Tool.  This 
appendix  takes  a  broad  view  of  automation  infrastructure  and  includes  automated  transaction 
processing  routines. 

KPP5:  NSPS  Automation  Infrastructure 

KPP  5.  NSPS  must  be  supported  by  a  robust  infrastructure  that  facilitates  user  operational  and 
functional  requirements.  That  infrastructure  must  include  an  interoperable  information 
technology  (IT)  system  that  is  easy  to  use;  accessible  to  all  users  with  appropriate  permissions; 
and  capable  of  generating  the  reports,  analyses  and  deliverables  necessary  for  all  types  of 
finance,  manpower  management,  HR  and  other  functional  requirements  and  for  evaluation  of 
the  NSPS.  A  second  element  of  the  NSPS  infrastructure  is  a  pay  and  performance  management 
information  system  that  provides  sufficiently  robust  data,  reports,  and  transaction  processing  to 
support  sound,  fair  compensation  decisions.  The  attributes  of  this  KPP  are  as  follows: 

■  Supporting  infrastructure  provides  interoperability  across  all  offices  and  functions 

■  Data  is  accessible  when  personnel  possess  appropriate  permissions 

NSPS  automation  infrastructure  covers  a  range  of  purposes.  One  set  comprises  the  tools 
necessary  for  human  resource  and  associated  functional  needs  including  personnel  transaction 
processing,  electronic  personnel  records  and  data  bases,  and  NSPS  conversion  tools.  The 
second  set  includes  the  tools  necessary  to  support  pay  and  performance  management 
processes.  Automation  instituted  in  Spiral  1  for  the  first  set  of  requirements  has  been  effective, 
if  unheralded,  being  mainly  behind-the-scenes  and  in  the  hands  of  practitioners.  Automation 
for  performance  management  has  been  in  the  forefront  for  most  employees,  supervisors,  and 
managers.  Problems  with  the  Performance  Appraisal  Application  for  employee  performance 
plans,  assessments,  and  ratings  have  been  pervasive  through  the  period  of  this  evaluation.  Pay 
pool  support  tools  have  been  reasonably  effective  and  useful.  Market  pay  reference  tools  have 
not  been  available  until  very  recently. 

The  first  set  of  automation  tools  includes: 

■  Mass  conversion  processing  routines  for  Spiral  employee  records  from  the  previous 
system  to  NSPS.  Effective  from  the  start  with  Spiral  1.1  (successful  conversion  rate  of 
98.7%,  up  99.1%  by  Spiral  2.2). 

■  Individual  transaction  processing  routines  and  codes  such  as  for  appointments, 
reassignments,  and  performance  payouts.  Code  structure  and  computer  tables  and 
routines  effective  from  the  start  with  Spiral  1.1.  No  serious  problem  reports  from  the  field 
regarding  life  cycle  actions  for  NSPS  employees;  bonuses  and  payouts  were  completed  on 
time  and  with  greater  than  99%  accuracy. 

■  The  NSPS  Readiness  Tool  to  support  component,  command/agency,  and  unit  level  NSPS 
transition  managers  in  managing  Spiral  conversions,  sharing  information  and  lessons 
learned,  and  monitoring  conversion  readiness.  Effective  from  the  start  with  Spiral  1.1. 
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■  NSPS  conversion  calculator  to  let  employees  find  out  what  they  can  expect  their  pay  to  be 
after  conversion  to  NSPS.  Effective  from  the  start  with  Spiral  1.1. 

The  second  set  of  tools  includes: 

■  Performance  Appraisal  Application  (PAA).  The  first  version  was  released  near  the  end  of 
the  first  Spiral  1.1  cycle.  It  was  difficult  to  use,  and  not  all  Spiral  1.1  organizations  chose  to 
use  it.  An  improved,  but  still  problem-ridden  second  version  was  implemented  near  the 
end  of  the  2007  performance  period,  just  in  time  for  performance  assessments.  In 
response  to  users,  a  third  version  has  been  developed  to  fix  remaining  problems  and 
meet  user  functionality  needs.8 

■  Compensation  Workbench  (CWB)  to  support  pay  pool  deliberations.  Effective  for  Spiral 
1.1  and  subsequently  modified  to  add  features  and  functionality. 

■  Pay  Pool  Analysis  Tool  (PAT)  to  assist  pay  pool  managers  and  Performance  Review 
Authorities  review  and  analyze  pay  pool  results  singly  or  in  groups  across  an  organization. 
Effective  when  introduced  for  the  second  performance  period. 

■  NSPS  market  pay  data  tools  to  provide  occupational  salary  and  wage  data  from  business 
establishments,  and  from  the  Federal  General  Schedule  sector,  to  compensation  analysts 
who  advise  on  pay  decisions.  Made  available  late  in  FY2008  beyond  the  time  frame  of  this 
evaluation. 


8 

These  actions  were  taken  to  get  user  feedback  and  recommendations  on  PAA: 

■  CPMS  conducted  PAA  and  Pay  Pool  Lessons  Learned  sessions  with  all  Component  agencies  immediately  after  the 
2008  payouts 

■  Usability  experts  involved  in  the  PAA  v2.0  testing  and  evaluation 

■  Usability  testing  with  pre-  and  post-testing  questionnaires  for  PAA  v2.0  and  3.0  completed  by  all  testers 

■  "Feedback"  link  added  to  PAA  2.0  to  collect  customer  comments  and  recommendations;  all  recommendations 
were  included  in  considering  PAA  v3.0  enhancements 

■  Usability  experts  involved  in  all  aspects  of  PAA  v3.0  design,  development,  and  testing 

■  Component  usability  test  event  conducted  in  June  2008  to  validate  PAA  3.0  design  with  users;  feedback 
incorporated  into  final  design  and  development 

■  Component/agency  testing  in  July/August  2008  prior  to  user  testing  for  PAA  v3.0 

■  Use  case  scenarios  used  by  testers  in  September  2008  to  ensure  end-to-end  testing  and  performance  of  PAA  3.0; 
testers  provided  pass/fail  results  for  every  item  in  each  scenario 

■  DoD's  standard  "ICE"  feedback  links  added  to  PAA  v3.0  to  garner  evaluation  and  comments  on  the  enhanced 
application 
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